17
$\begingroup$

Background

I have a variable with an unknown distribution.

I have 500 samples, but I would like demonstrate the precision with which I can calculate variance, e.g. to argue that a sample size of 500 is sufficient. I am also interested in knowing the minimum sample size that would be required to estimate variance with a precision of $X\%$.

Questions

How can I calculate

  1. the precision of my estimate of the variance given a sample size of $n=500$? of $n=N$?
  2. How can I calculate the minimum number of samples required to estimate the variance with a precision of $X$?

Example

Figure 1 density estimate of the parameter based on the 500 samples.

enter image description here

Figure 2 Here is a plot of sample size on the x-axis vs. estimates of variance on the y axis that I have calculated using subsamples from the sample of 500. The idea is that the estimates will converge to the true variance as n increases.

However, the estimates are not valid independent since the samples used to estimate variance for $n \in [10,125,250,500]$ are not independent of each other or of the samples used to calculate variance at $n\in [20,40,80]$

enter image description here

$\endgroup$
  • $\begingroup$ Just be aware that if a component of your unknown distribution is a Cauchy distribution, the variance is undefined. $\endgroup$ – Mike Anderson Feb 9 '11 at 12:18
  • $\begingroup$ @Mike Or indeed an infinite number of other distributions. $\endgroup$ – Glen_b Sep 11 '16 at 5:22
10
$\begingroup$

For i.i.d. random variables $X_1, \dotsc, X_n$, the unbiased estimator for the variance $s^2$ (the one with denominator $n-1$) has variance:

$$\mathrm{Var}(s^2) = \sigma^4 \left(\frac{2}{n-1} + \frac{\kappa}{n}\right)$$

where $\kappa$ is the excess kurtosis of the distribution (reference: Wikipedia). So now you need to estimate the kurtosis of your distribution as well. You can use a quantity sometimes described as $\gamma_2$ (also from Wikipedia):

$$\gamma_2 = \frac{\mu_4}{\sigma_4} - 3$$

I would assume that if you use $s$ as an estimate for $\sigma$ and $\gamma_2$ as an estimate for $\kappa$, that you get a reasonable estimate for $\mathrm{Var}(s^2)$, although I don't see a guarantee that it is unbiased. See if it matches with the variance among the subsets of your 500 data points reasonably, and if it does don't worry about it anymore :)

$\endgroup$
  • $\begingroup$ do you have a textbook reference for the unbiased estimator of variance? I don't know where to go from Wikipedia for more context. $\endgroup$ – Abe Feb 9 '11 at 20:02
  • $\begingroup$ I don't have my standard text Rice with me here, so I can't check the page number for you, but I'm sure it's in there. Wikipedia suggests it should also be mentioned in: Montgomery, D.C. and Runger, G.C.: Applied statistics and probability for engineers, page 201. John Wiley & Sons New York, 1994. $\endgroup$ – Erik P. Feb 9 '11 at 20:21
  • $\begingroup$ thanks for your help with this. This answer has been very useful and it has been informative to quantify variance uncertainty - I have applied the equation about 10 times in the last day. calculating $kappa$ is easy with the moments library: library(moments); k <- kurtosis(x); n <- length(x); var(x)^2*(2/(n-1) + k/n) $\endgroup$ – Abe Feb 10 '11 at 16:50
  • $\begingroup$ any chance you found the page number from the Rice text? I can't find it in Casella and Berger. A primary reference would be even better if you know it. The wikipedia page is notably un-referenced. $\endgroup$ – Abe Jun 5 '12 at 22:35
  • $\begingroup$ Hmmm... looks like Rice doesn't have the formula either. I'll keep an eye out for it, but at this point I don't have a reference at all. $\endgroup$ – Erik P. Jun 6 '12 at 0:04
15
$\begingroup$

Learning a variance is hard.

It takes a (perhaps surprisingly) large number of samples to estimate a variance well in many cases. Below, I'll show the development for the "canonical" case of an i.i.d. normal sample.

Suppose $Y_i$, $i=1,\ldots,n$ are independent $\mathcal{N}(\mu, \sigma^2)$ random variables. We seek a $100(1-\alpha)\%$ confidence interval for the variance such that the width of the interval is $\rho s^2$, i.e., the width is $100\rho \%$ of the point estimate. For example, if $\rho = 1/2$, then the width of the CI is half the value of the point estimate, e.g., if $s^2 = 10$, then the CI would be something like $(8,\,13)$, having a width of 5. Note the asymmetry around the point estimate, as well. ($s^2$ is the unbiased estimator for the variance.)

"The" (rather, "a") confidence interval for $s^2$ is $$ \frac{(n-1) s^2}{\chi_{(n-1)}^{2\;(1-\alpha/2)}} \leq \sigma^2 \leq \frac{(n-1) s^2}{\chi_{(n-1)}^{2\;(\alpha/2)}} \>, $$ where $\chi_{(n-1)}^{2\;\beta}$ is the $\beta$ quantile of the chi-squared distribution with $n-1$ degrees of freedom. (This arises from the fact that $(n-1)s^2/\sigma^2$ is a pivotal quantity in a Gaussian setting.)

We want to minimize the width so that $$ L(n) = \frac{(n-1) s^2}{\chi_{(n-1)}^{2\;(\alpha/2)}} - \frac{(n-1) s^2}{\chi_{(n-1)}^{2\;(1-\alpha/2)}} < \rho s^2 \>, $$ so we are left to solve for $n$ such that $$ (n-1) \left(\frac{1}{\chi_{(n-1)}^{2\;(\alpha/2)}} - \frac{1}{\chi_{(n-1)}^{2\;(1-\alpha/2)}} \right) < \rho . $$

For the case of a 99% confidence interval, we get $n = 65$ for $\rho = 1$ and $n = 5321$ for $\rho = 0.1$. This last case yields an interval that is (still!) 10% as large as the point estimate of the variance.

If your chosen confidence level is less than 99%, then the same width interval will be obtained for a lower value of $n$. But, $n$ may still may be larger than you would have guessed.

A plot of the sample size $n$ versus the proportional width $\rho$ shows something that looks asymptotically linear on a log-log scale; in other words, a power-law--like relationship. We can estimate the power of this power-law relationship (crudely) as

$$ \hat{\alpha} \approx \frac{\log 0.1 - \log 1}{\log 5321 - \log 65} = \frac{-\log 10}{\log \frac{5231}{65}} \approx -0.525 , $$

which is, unfortunately, decidedly slow!


This is sort of the "canonical" case to give you a feel for how to go about the calculation. Based on your plots, your data don't look particularly normal; in particular, there is what appears to be noticeable skewness.

But, this should give you a ballpark idea of what to expect. Note that to answer your second question above, it is necessary to fix some confidence level first, which I've set to 99% in the development above for demonstration purposes.

$\endgroup$
  • $\begingroup$ this is a very nice answer to my question. However, although I follow the calculation that you make for $n|\rho$, it is not exactly clear to me if the units for $rho$ is percent in the solution $n=65$ for $\rho<1$; does this mean "$\rho$ is less than $1\times s^2$" or "$\rho$ less than $1\%$ of $s^2$? $\endgroup$ – Abe Feb 9 '11 at 5:58
  • $\begingroup$ @Abe, updated and hopefully clarified in the process. There was one particularly bad typo in the previous version. Sorry about that. $\endgroup$ – cardinal Feb 9 '11 at 12:55
  • $\begingroup$ a very nice answer, but I chose the one from @Erik because it is more applicable to my problem (as my parameter is not normally distributed). $\endgroup$ – Abe Jun 5 '12 at 22:37
  • $\begingroup$ @Abe: Not a problem. That is what the checkmark is there for. My answer was (is) intended to be illustrative, more than anything. From what I can tell, it does still appear to be the only one that addresses both of your questions, and will be (asymptotically) correct even in the scenario that Erik outlines. (+1 to him well over a year ago.) :) $\endgroup$ – cardinal Jun 6 '12 at 0:40
  • $\begingroup$ You are correct and I am glad that I have now revisited your answer. I had ended up using the general calculation by @Erik, but now I see the value in the general solution. Plus, presenting the CI rather than SD will solve an issue with my audience being confused when seeing a statistic in the form of $s(s_{s})$, not understanding what the variance of a variance is. So $s[lcl,ucl]$ should make this more clear, and consistent with other statistical summaries. And it will be helpful to show asymmetry. $\endgroup$ – Abe Jun 6 '12 at 14:28
1
$\begingroup$

I would focus on the SD rather than the variance, since it's on a scale that is more easily interpreted.

People do sometimes look at confidence intervals for SDs or variances, but the focus is generally on means.

The results you give for the distribution of $s^2/\sigma^2$ can be used to get a confidence interval for $\sigma^2$ (and so also $\sigma$); most introductory math/stat texts would give the details in the same section in which the ditribution of $\sigma^2$ was mentioned. I would just take 2.5% from each tail.

$\endgroup$
  • $\begingroup$ (This reply came here after a duplicate question, framed somewhat differently, was merged.) $\endgroup$ – whuber Oct 18 '11 at 20:11
1
$\begingroup$

The following solution was given by Greenwood and Sandomire in a 1950 JASA paper.

Let $X_1,\dots,X_n$ be a random sample from a $\mathrm{N}(\mu,\sigma^2)$ distribution. You will make inferences about $\sigma$ using as (biased) estimator the sample standard deviation $$ S=\sqrt{\sum_{i=1}^n\frac{(X_i-\bar{X})^2}{n-1}}, $$ and you want to control the probability that the relative deviation between $S$ and $\sigma$ is within a fraction $0<u<1$. That is, $$ \Pr\{S<(1-u)\cdot\sigma\}=a \quad\text{and}\quad \Pr\{S>(1+u)\cdot\sigma\}=b, $$ in which the significance level $\gamma=1-a-b$.

It follows that $$ \Pr\!\left\{ \frac{(n-1)S^2}{\sigma^2} < (n-1)(1-u)^2\right\} = a $$ and $$ \Pr\!\left\{ \frac{(n-1)S^2}{\sigma^2} > (n-1)(1+u)^2\right\} = b. $$ Since the pivotal quantity $(n-1)S^2/\sigma^2$ has $\chi^2_{n-1}$ distribution, adding the two probabilities, we find

$$ \gamma = F_{\chi^2_{(n-1)}}((n-1)(1+u)^2) - F_{\chi^2_{(n-1)}}((n-1)(1-u)^2), $$

and the necessary sample size is found solving the former equation in $n$ for given $\gamma$ and $u$.

R code.

gamma <- 0.95
u <- 0.1
g <- function(n) pchisq((n-1)*(1+u)^2, df = n-1) - pchisq((n-1)*(1-u)^2, df = n-1) - gamma
cat("Sample size n = ", ceiling(uniroot(g, interval = c(2, 10^6))$root), "\n")

Output for $u=10\%$ and $\gamma=95\%$.

Sample size n = 193
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.