Originally Posted by
David Franklin
To be honest, I'm suspecting it's more incompetence than anything else.
Part of what I do in my job is processing satellite data. I've been doing it for 10 years now. The datasets have always been "large" relative to the computing facilities I've had. And when you process the data, there's usually 5 or 6 'temporary' stages, all of which you need to store, at least temporarily.
And then when you're finished, it's the processed data that you work with all the time, not the original. (That's why you processed it, after all). So that's the data that's "live" on the hard drives, that gets backed up, distributed to other users etc.
Of course, the original data should be backed up as well, but that is usually passed onto the work of the system administrators / general dogsbodies. And (if they were made at all), those backups were probably made 10 years ago, and never checked.
When you also factor in that the staff (particularly the sysadmin staff) tend to come and go in these places, it's quite likely that it's impossible to find where the stuff was actually backed up to.
True personal story: So, we found a possible issue in some data that I processed something like 6 years ago. And our sysadmin person (who was responsible for backing them up) had just left the company - having said, "oh, everything's in the cupboard over there". I was then left searching through something like 500 badly labelled DVDs and external hard drives to try to find the data. Only to find out he'd backed up the wrong stuff!
But, and it's a big but. I'm pretty paranoid about keeping copies of the original data. So I got out my personal backup stash of 300 DVDs, and I found the relevant DVDs. Only 4 of the 5 DVDs worked (not a perfect backup medium, in my experience), but they had the data I needed.
I think it's pretty inexcusable that the researchers didn't safeguard the original data. But I do suspect it was more laziness/incomptence than anything else. (Particularly if it was originally on paper/magnetic tape. I can just see them getting a new computer system and then 2 years later realising "uh, you know that data on paper tape - we don't have anything that reads it any more". So the data on paper tape gets put into a warehouse of data on paper tape that's kept "just in case", and then 5 years later someone else says "what's the point of a warehouse full of stuff we never look at" and throws it all away).
Bookmarks