Let's say I have an histogram of some continuous data with unknown and arbitrary pdf. I don't have access to the original raw data. So I don't know that 3 of my data points are 21, 25 and 27, all I know is that 3 of my data points are in the bin 20-30. How accurate is it to compute the moments of this pdf from the binned data? Can I estimate the error I'm making?
$\begingroup$
$\endgroup$
1
-
2$\begingroup$ Closely related: Standard deviation of binned observations. $\endgroup$– whuber ♦Sep 19, 2013 at 16:16
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
See Experimental uncertainty estimation and statistics for data having interval uncertainty (Ferson et al. 2007). Calculating the uncertainty in the mean is easy, just compute the mean of the data assuming all points are at the lowest value of the interval and the mean assuming all points are at the highest value of the interval. The true mean has to lie somewhere in this interval.
Variance is a bit harder. But the referenced publication gives some algorithms to estimate it, and fortunately for binned histogram data it is a case of what Ferson et. al (2007) refer to as no-nesting type data.
There are various other measures that are easy to calculate by only examining the end points of the intervals (like the CDF). Ferson et. al (2007) refer to such measures as stochastic dominant (wording taken from Charles Manski).
-
$\begingroup$ Thanks a lot. Just skimmed through it briefly, and indeed since the binned data is "skinny" and "non-nesting", I can get some rough estimates pretty easily. And if I want to dig deeper, this reference will give me plenty of options. $\endgroup$ Sep 19, 2013 at 13:50
-
3$\begingroup$ (+1) (I have worked with Scott Ferson and enjoyed many stimulating conversations with him about this subject.) It is worth understanding that the bounds given in the literature you cite are extreme: they correspond to arbitrary worst distributions. If you have any expectations about the underlying distribution at all, even something as mild as (say) having a continuous PDF and no more than three modes, you can always obtain better bounds with standard semi-parametric or Bayesian techniques. $\endgroup$– whuber ♦Sep 19, 2013 at 16:15