Ok, fair warning--this is a philosophical question that involves no numbers. I've been thinking a lot about how errors creep into data sets over time and how that should be treated by analysts--or if it should really matter at all?
For background, I'm doing the analysis on a long-term study that involves many data sets collected by probably 25 people over 7-8 years--nobody has ever brought all the data into a coherent structure (that's my job). I've been doing a lot of data-entry (transcribing from photocopies of old lab notebooks) and I keep finding small transcription errors that other folks made, and also finding data entries that are difficult or impossible to read--mostly because the ink has faded over time. I'm using context to make 'best guesses' about what the data says and leaving the data point out altogether if I'm not fairly certain. But I keep thinking about the fact that every time data is copied, the frequency of errors will inevitably increase until the original data is completely lost. (This is akin to copying a movie to a VHS tape, then using the copy to make another copy, and repeating the process over and over until all you get is random noise and static on the screen.)
So, this leads me to a thought: in addition to instrument/measurement errors, and recording errors, there is a fundamental 'data handling error' component that will increase over time and with more handling of the data (side note: this is probably just another way of stating the 2nd law of Thermodynamics, right? Data entropy will always increase). Consequently, I wonder if there should be some kind of 'correction' introduced to account for the life-history of data sets (something akin to a Bonferroni correction)? In other words, should we assume that older, or more copied data sets are less accurate, and if so, should we adjust findings accordingly?
But then my other thought is that errors are an inherent part of data collection and data handling, and since all the statistical tests have been developed with real-world data, perhaps these sources of error are already 'priced in' to the analysis?
Also, another point worth mentioning is that since data errors are random, they are far more likely to reduce the strength of a finding than to improve it--in other words, data handling errors would lead to Type 2 errors, not Type 1 errors. So, in many contexts, if you were using old/questionable data and still found an effect, that would increase your confidence that the effect is real (because it was strong enough to survive the addition of random error to the data set). So for that reason, perhaps the 'correction' should go the other way (increase the alpha-level required for a 'finding'), or just not trouble us?
Anyway, sorry to be so verbose and obtuse, I'm not really sure how to ask this question more concisely. Thanks for bearing with me.