2
$\begingroup$

I have a large data set that I'd like to perform statistical analysis on, and I'm in the beginning stages of this process. This data set comes from a large sample of user inputs of integer values from 1-30(ish). I'd like to compute some basic statistics like mean and standard deviation, but I have an issue with the data. Because it comes from human input, people frequently select "round" numbers, such as 15 or 20, drastically skewing any results. Is there an elegant way to handle this problem? I'd imagine that I could perform some sort of averaging or smoothing on a histogram to approximate this, but that seems messy. I don't know of any process that handles something like this, but certainly someone has faced the problem before.

Thanks!

$\endgroup$
  • 1
    $\begingroup$ You could maximize a likelihood based on interval data, that is, assuming a datum 15 really means some interval, like$[12.5, 17.5]$. $\endgroup$ – kjetil b halvorsen Mar 9 '15 at 16:44
  • 3
    $\begingroup$ Census bureaus deal with this all the time, because people tend to round their ages to a multiple of 5: consult their documentation. What you do about it, though, depends on what kind of analysis you plan. Note that you are not describing anything specific to Normal distributions--mean and SD are merely descriptive statistics--and that "continuous integer values" is a contradiction in terms: integral values are the opposite of continuous. Indeed, the relatively small range suggests caution in using a continuous model (such as a Normal distribution) for any analysis. $\endgroup$ – whuber Mar 9 '15 at 18:16
  • $\begingroup$ @whuber: good call on the census bureau, I'll look into that. I edited the post to remove the continuous part, I was a little unsure of that when writing it and you're correct. It's a fairly small range but the physical effect of the value is pretty varied, and I have several thousand data points which I hope lends enough power to make valid inferences. $\endgroup$ – french13 Mar 9 '15 at 18:34
3
$\begingroup$

The phenomenon you describe is called "Number Preference". This article seems to answer your question: "Statistical Tests and Measures for the Presence and Influence of Digit Preference"

$\endgroup$
  • $\begingroup$ "Number Preference" is exactly what I was looking for. My hypothesis is that my distribution is normal and number preference is distorting the actual values of the normal distribution, but I think fixing that is perhaps a separate question, so I've accepted your edits. Thank you for your help. $\endgroup$ – french13 Mar 9 '15 at 20:26
2
$\begingroup$

As a reference for future readers of this question, this paper also contains a detailed explanation of methods for smoothing data containing digit preference: Modelling general patterns of digit preference

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.