10
$\begingroup$

Ok, fair warning--this is a philosophical question that involves no numbers. I've been thinking a lot about how errors creep into data sets over time and how that should be treated by analysts--or if it should really matter at all?

For background, I'm doing the analysis on a long-term study that involves many data sets collected by probably 25 people over 7-8 years--nobody has ever brought all the data into a coherent structure (that's my job). I've been doing a lot of data-entry (transcribing from photocopies of old lab notebooks) and I keep finding small transcription errors that other folks made, and also finding data entries that are difficult or impossible to read--mostly because the ink has faded over time. I'm using context to make 'best guesses' about what the data says and leaving the data point out altogether if I'm not fairly certain. But I keep thinking about the fact that every time data is copied, the frequency of errors will inevitably increase until the original data is completely lost. (This is akin to copying a movie to a VHS tape, then using the copy to make another copy, and repeating the process over and over until all you get is random noise and static on the screen.)

So, this leads me to a thought: in addition to instrument/measurement errors, and recording errors, there is a fundamental 'data handling error' component that will increase over time and with more handling of the data (side note: this is probably just another way of stating the 2nd law of Thermodynamics, right? Data entropy will always increase). Consequently, I wonder if there should be some kind of 'correction' introduced to account for the life-history of data sets (something akin to a Bonferroni correction)? In other words, should we assume that older, or more copied data sets are less accurate, and if so, should we adjust findings accordingly?

But then my other thought is that errors are an inherent part of data collection and data handling, and since all the statistical tests have been developed with real-world data, perhaps these sources of error are already 'priced in' to the analysis?

Also, another point worth mentioning is that since data errors are random, they are far more likely to reduce the strength of a finding than to improve it--in other words, data handling errors would lead to Type 2 errors, not Type 1 errors. So, in many contexts, if you were using old/questionable data and still found an effect, that would increase your confidence that the effect is real (because it was strong enough to survive the addition of random error to the data set). So for that reason, perhaps the 'correction' should go the other way (increase the alpha-level required for a 'finding'), or just not trouble us?

Anyway, sorry to be so verbose and obtuse, I'm not really sure how to ask this question more concisely. Thanks for bearing with me.

$\endgroup$
6
  • 7
    $\begingroup$ It's a great question (+1). One point, though: it could be a substantial error to treat most of the data errors you mention as "random." For instance, there tend to be far more interchanges of the digits "0", "5", "6", and "8" during transcriptions than of other digits (and some of these can be misread as "." and vice versa). Also, changes made to prominent data values (such as the extremes) are often quickly identified and fixed. Although there certainly is some element of chance to these data-corruption processes, characterizing them correctly can be an important issue. $\endgroup$ – whuber Dec 29 '14 at 18:55
  • 1
    $\begingroup$ Why don't you treat data handling errors are the part of measurement errors and deal with them accordingly? If in order to measure the number amusement park riders, I need to deploy 20 people to watch the gates, then I can consider this 20 people team as a measurement device of sorts $\endgroup$ – Aksakal Dec 29 '14 at 19:05
  • $\begingroup$ @whuber, it's still random to mix up 8 and 5, though it may not have equal probability with mixing up 5 and 7. $\endgroup$ – Aksakal Dec 29 '14 at 19:07
  • 1
    $\begingroup$ @whuber, that's a fascinating point (non-equal frequency of certain types of transcription errors) that I hadn't thought about. Can you point me toward any sources to learn more about that? It makes me wonder if a data-quality test could be developed, based on digit frequency? I've heard of similar tests for fraudulent/faked data based on digit frequency, so I imagine something similar would be possible if the trends you mention are consistent. $\endgroup$ – Jas Max Dec 29 '14 at 19:16
  • $\begingroup$ @whuber, one more thought. You mention 0, 5, 6, 8 are often confused--because they look alike? It makes me realize that different sources of error would have characteristic substitution errors--for example, if you were hearing the data (recording what somebody said) then I think 5 and 9 would probably be more frequently confused. If the source of error was entropy (ink fading or electrons moving) then I think the substitution would be more random, but possibly also unique. If these patterns held,perhaps you could investigate sources of error in large data sets, based on digit frequency. $\endgroup$ – Jas Max Dec 29 '14 at 19:53
3
$\begingroup$

I second the suggestion of @Aksakal: If measurement error is seen by the analyst as potentially important, it can and should be modeled explicitly as part of the data-generating process.

I see several considerations that argue against the introduction of a generic correction factor based on, e.g., the age of the data set.

First, age may be a very poor proxy for the degree of data deterioration. The technology of duplication, compression, and conservation, and the degree of effort and care that went into verifying correct transcription, are apparently the important factors. Some ancient texts (e.g., The Bible) have been conserved for centuries with apparently zero degradation. Your VHS example, while legitimate, is actually unusual, in that each duplication event always introduces error, and there are not easy ways to check for and correct for transcription errors -- if one uses cheap, widely available technologies for duplication and storage. I expect that one lower the degree of introduced errors substantially, through investments in more expensive systems.

This last point is more general: data conservation and propagation are economic activities. The quality of transmission depends greatly on the resources deployed. These choices will in turn depend on the perceived importance of the data to whoever is doing the duplicating and transmitting.

Economic considerations apply to the analyst, as well. There are always more factors you can take into account when doing your analysis. Under what conditions will data transcription errors be substantial enough, and important enough, that they are worth taking into account? My hunch is: such conditions are not common. Moreover, if potential data degradation is seen as important enough to account for it in your analysis, then it is probably important enough to make the effort to model the process explicitly, rather than insert a generic "correction" step.

Finally, there is no need to develop such a generic correction factor de novo. There exists already a substantial body of statistical theory and practice for analyzing data sets for which measurement error is seen as important.

In sum: it's an interesting thought. But I don't think it should spur any changes in analytic practice.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.