Let $X_1, X_2, \dotsc, X_k$ be an i.i.d. sample of a random variable $X$. I plot these in a histogram and would like to include a confidence interval for the height of each histogram bar. Do you know how to go about doing it?
$\begingroup$
$\endgroup$
1
-
3$\begingroup$ How were the bins chosen? It's straightforward to obtain CIs for the heights when the cutpoints are chosen independently of the data, but otherwise it's much more challenging. Normally, bins for histograms are chosen on the basis of data characteristics, including their amount, their range, whether there are apparent outliers, and how skew they might be: this creates a lack of independence which is difficult to quantify (and therefore it ought to color our interpretation of any CIs we might attempt to calculate for the heights). You can avoid these complications by using a probability plot. $\endgroup$– whuber ♦May 25, 2012 at 4:28
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
5
Let there be $i \in 1, \dots, I$ histogram bins. The probability of falling into a particular bin is $p_i$. This is just a binomial trial (i.e., you are either in the bin or not, each with a given probability).
If you are calculating the frequency of being in the bin (i.e., histograms with bars giving the set of $p_i$'s), then the variance should be $p_i(1-p_i)/k$.
If you are calculating total counts, then the variance is $p_i(1-p_i) \times k$.
The confidence interval can then be formed in the standard way.
-
$\begingroup$ Let say I have 100,000 samples. 30,000 of them fall into bin 1 (e.g. values 0-100). Then p_1 = 0.3 and variance = p_1*(1-p_1)/100,000 = 0.3*0.7/100,000? $\endgroup$ May 25, 2012 at 2:24
-
-
1$\begingroup$ Shouldn't the variance rather be 100,000*0.3*0.7? $\endgroup$ May 25, 2012 at 2:29
-
$\begingroup$ Ahh, good point @JohnManak, I usually plot frequencies as in proportions. I'll make it clearer. $\endgroup$– CharlieMay 25, 2012 at 2:35
-
$\begingroup$ Notice that the histogram is a biased estimator of the density. Hence, the interval computed from the binomial distribution is a confidence interval for the "histogramized" density rather than the density itself. This is explained in Wasserman, "All of nonparametric statistics", 2006, Springer page 130. Also notice that 1) this is a pointwise confidence interval (not a confidence band) and 2) it cannot be computed this way when the density estimate is zero because $p_i$ is estimated as zero. $\endgroup$ Dec 17, 2022 at 18:50
$\begingroup$
$\endgroup$
Using the binomial variance $p (1-p) k$, as proposed in another answer, is only a good idea when the proportion is not near 0 or 1.
For better-behaved confidence intervals, there is extensive statistical literature e.g. Agresti & Coull (1998). Some of the proposed formulae are implemented in the R library PropCIs. Here's an example creating a histogram with error bars using PropCIs: Errorbars on histograms
answered May 28, 2013 at 23:47