7
$\begingroup$

Suppose I have $N$ samples each of size $n$, drawn from the same population, where each sample has its own sample variance $s_i^2$.

I understand that for any given sample, a first estimate of the population variance $\sigma^2$ is:

$$\hat{\sigma}^2 = s_i^2 \displaystyle\frac{n}{n-1}$$

But what if you have multiple samples? What is the best estimate of $\sigma^2$ given multiple $s^2$?

$\endgroup$
6
  • $\begingroup$ Some meta-analysis techniques might be of interest. $\endgroup$
    – Noah
    Nov 3 '16 at 4:46
  • $\begingroup$ Could you explain why you multiply the sample variance by $(n-1)/n$? There's nothing illegitimate about it, but your reason for including that factor will shed light on what you might mean by "best" estimate of $\sigma^2$. $\endgroup$
    – whuber
    Nov 8 '16 at 16:14
  • 1
    $\begingroup$ @whuber I made a mistake, the factor should have been $n/(n-1)$. The factor makes it an unbiased estimator of the population variance, according to mathworld.wolfram.com/SampleVariance.html $\endgroup$
    – Delyle
    Nov 16 '16 at 18:36
  • $\begingroup$ In that case the answer you have accepted is completely wrong, understanding you want an unbiased estimate of the variance of the population. The expected value of the accepted answer is $N$ times larger than it should be. It is also inadmissible in the sense that other unbiased estimates can be constructed that have smaller standard errors. $\endgroup$
    – whuber
    Nov 16 '16 at 18:38
  • $\begingroup$ @whuber Thank you for pointing that out. Do you have a proposed solution? Also it would probably be best to comment on the answer as well/instead $\endgroup$
    – Delyle
    Nov 16 '16 at 18:42
8
$\begingroup$

A comment clarified that the best estimator of the variance is intended to be unbiased. A standard way to find such an estimator is to restrict one's attention to linear combinations of the estimators (because almost anything else would be difficult to analyze).

Let there be $k$ independent samples indexed by $i=1,2,\ldots, k$, each with its own unbiased variance estimator $\hat\sigma_i^2$. Let the unknown weights of the linear combination be $w_i$, so that the combined estimator will be

$$\hat\sigma^2 = \sum_{i=1}^k w_i \hat \sigma_i^2.$$

Because this is supposed to be unbiased for any population, by definition the population variance will equal its expected value:

$$\sigma^2 = \mathbb{E}(\hat\sigma^2) = \sum_{i=1}^k w_i \mathbb{E}(\hat\sigma_i^2) = \sum_{i=1}^k w_i \sigma^2 = \left(\sum_{i=1}^k w_i\right)\sigma^2.$$

Since $\sigma^2 \ne 0$ is possible, division of both sides by $\sigma^2$ implies the weights sum to unity:

$$1 = \sum_{i=1}^k w_i.$$

Let the sample size for estimator $\hat\sigma_i$ be $n_i$. (In the question all the $n_i$ are equal to $n$.) Because each estimator $\hat\sigma_i^2$ has $n_i-1$ degrees of freedom, its variance will be approximately proportional to $1/(n-1)$ times some value that is an (unknown) property $f$ of the population. (This unknown property depends on the first four moments of the population.) In fact, for a Normal population this is not an approximation at all: the variances of the estimators are exactly proportional to $1/(n_i-1)$. Therefore we may approximate the variance of the combined estimator as

$$\operatorname{Var}(\hat\sigma^2) = \operatorname{Var}\left(\sum_{i=1}^k w_i \hat\sigma_i^2\right) = \sum_{i=1}^k w_i^2 \operatorname{Var}(\hat\sigma_i^2) \approx \sum_{i=1}^k w_i^2 \frac{f}{n_i-1}.$$ The second equality is due to the independence of the $k$ samples.

Subject to the sum-to-unity constraint, this variance is minimized when the $w_i$ are proportional to $n_i-1$. Therefore an (approximate) minimum variance unbiased linear estimator of the population variance is

$$\hat\sigma^2 = \frac{1}{n_1+\cdots n_k - k}\sum_{i=1}^k (n_i-1)\hat\sigma_i^2.$$

When all the $n_i$ are equal to a common value $n$, this reduces to the arithmetic mean of the individual variance estimators. It should be intuitively obvious that some kind of equally-weighted average of all $k$ estimators would be the best one in this case.

$\endgroup$
3
  • $\begingroup$ +1 Very clear explanation. How does this estimate compare to pooling observations from all the samples and using the sample variance? That is, to consider all the observations as one large sample and calculate the sample variance. Is it equivalent? $\endgroup$ Mar 17 at 12:20
  • $\begingroup$ Of course, if each sample is drawn from a population with a different mean (but same variance), then the sample variance of the pooled observations would not be valid. But if the means are identical, then I think your estimator should be equivalent to the sample variance of the pooled observations? $\endgroup$ Mar 17 at 12:44
  • 1
    $\begingroup$ @Anomaly I wouldn't go quite so far as to say "not valid," but you're certainly right to call into question what such a statistic might mean and to wonder what it could be estimating. Such calculations come into their own in situations where it's possible the subsamples do come from a common population and your objective is to test that hypothesis. My estimator is not the same as the sample variance of the pooled observations, though, as you can see by considering a situation with two subgroups of two observations each and each subgroup has a tiny variance but their means differ greatly. $\endgroup$
    – whuber
    Mar 17 at 13:26
0
$\begingroup$

How about putting a hat on your estimates of the variance for each of the N samples

$\hat{\sigma}_{i}^{2}\simeq s_{i}^{2}\frac{n-1}{n}$

and appealing to the Law of Propagation of Uncertainty, add them in quadrature to get the estimate of the overall variance

$\tilde{\sigma}^{2}=\sum_{i=1}^{N}\hat{\sigma}_{i}^{2}$

$\endgroup$
2
  • $\begingroup$ It looks like your formula should not have any square root and, regardless, it will grossly overestimate the variance. Try it out on some simple data to see what's going wrong. $\endgroup$
    – whuber
    Nov 8 '16 at 16:12
  • $\begingroup$ I made an initial mistake which has been copied here, the factor should be $n/(n-1)$ and not the inverse. @whuber Have the issues you brought up been fixed? $\endgroup$
    – Delyle
    Nov 16 '16 at 18:38
0
$\begingroup$

I presume this is the issue where you have k samples size n of a population that are all random with a given mean m and sample unbiassed variance s^2. As you do not have the real population mean, NOR the population variance that is used in all the textbooks, you have to find a means to not only calculate this population mean (rather easy), but also have a good estimator of the population variance. I did not find a description how to do this in the HANDBOOK OF PARAMETRIC AND NONPARAMETRIC STATISTICAL PROCEDURES (2000 pages, really quite complete) and that leaves us two ways. Theory (approach each sample of the same parameter as RANDOM and thus state the population is a sum of the INDEPENDENT samples with a weight 1/k of each, giving you a resulting population variance if you pull another sample size n. Then multiply that population variance with n to obtain the population variance when pulling single size samples). Another way is to pragmatically create a program that simulates your population (does not have to be exact) to calculate variances of many sample sizes using your guessed formula and then see what method is actually (after 1000 repetitions, computers are patient) most robust. I propose you use the theoretical approach.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.