1
$\begingroup$

I have a number of samples of sample size 2 and a number of sample of sample size 3. If my samples are all samples from populations with a shared population variances, I wish to estimate population variance. I wish to estimate population variance in the way that will give me a value as close as theoretically possible (with the information I possess) to the true value of population variance.

While struggling with the question of how the sample numbers (planning to average corrected values) affect the sample variance corrections I should be using (because increased sample number of averaged sample values decreases the average's variance without decreasing its bias), I came across another concept I was unfamiliar with.

In the accepted answer to Maximum Likelihood Estimation — why it is used despite being biased in many cases, it is said that the corrected denominator for an unbiased estimator is $n-1$ (which I knew), that the corrected denominator for the minimum mean-squared error estimator is $n+1$ (which I knew), and that the corrected denominator for the maximum likelihood estimator is $n$ (which I did not know).

If I understand correctly, each of these is corrected denominator for when estimating a population variance from a single sample (rather than an average of values from multiple samples), thus I cannot use them directly in any case.

Reiterating, I wish to use the approach which will give me the value theoretically closest to the true population variance value, given the information available to me. I thought this would be the approach which gives the minimum mean squared error ('getting arrows clustered as closely to the bullseye as possible'), but then how can the maximum likelihood estimator (the value which is most likely to give rise to the observed data) be different?

To rephrase, given that I want to get a final value which is close as possible to the true value (of the population variance), which of these approaches--MMSE and MLE--is the wrong approach, and why?

(Of course, if there is yet another approach which serves my intention even better that I haven't heard about yet I would be glad to learn of it!)

$\endgroup$
  • $\begingroup$ @BruceET Thank you for your examples! If I understand correctly, this agrees with my impression that $(n-1)$ looks best for an estimation from averaging many measurements, while $(n+1)$ is best for from just one measurement, and in-between values appropriate for averaging a few measurements. When writing my question I intended to specifically ask about why the Maximum Likelihood Estimation would have a denominator which was $(n)$ rather than $(n+1)$ (or $(n-1)$). I would also like to ask now what the mathematical meaning of $^\ast2$ is in this context, as I am unfamiliar with this! $\endgroup$ – MCC Nov 18 '19 at 16:47
  • $\begingroup$ --Hm. Attempting to recreate the symbol you used according to here seems to have failed. May I also ask how you wrote it? $\endgroup$ – MCC Nov 18 '19 at 16:49
  • 1
    $\begingroup$ Sorry, it should be $E(S^2) = \sigma^2.$ // The sample size $n > 1$ would not ordinarily dictate whether to use unbiased estimator with denominator $n - 1$ or estimator with least mean sq error with $n+1$ in denom. (normal data)// MLE results from a mathematical computation; it has $n$ in denominator because that is the mathematical result. // The estimator that gets 'as close as possible' depends on the distribution of the population (normal, exponential, Laplace, etc.) and how you measure 'close' . If you have $n = 3$ observations from a normal population, then see the addendum to my answer. $\endgroup$ – BruceET Nov 18 '19 at 17:59
  • $\begingroup$ If I understand correctly, 'n' refers to the sample size of a single sample, whereas a different variable (what?) would be used for the sample number when averaging the corrected sample variances of many samples. For infinite sample number, variance of the mean-corrected-sample-variance shrinks to 0 while bias stays the same, so you want 0 bias; for sample number 1 both variance and bias are high, so you want the correction that gives the lowest MSE for a single sample. // Why does MLE give that mathematical result rather than an $n+1$ mathematical result? $\endgroup$ – MCC Nov 18 '19 at 18:07
0
$\begingroup$

Comment continued:

Here is a brief simulation with standard normal data, for which $\sigma^2 = 1.$ In the simulation v1 is a vector a million unbiased estimates $S^2,$ each for a sample of size $n = 10.$

set.seed(1116)
m = 10^6; n = 10; 
x = rnorm(m*n); DTA = matrix(x, nrow=m)

v1 = apply(DTA, 1, var)  # denom n-1
mean(v1);  mean((v1-1)^2)
[1] 0.9998608  # aprx 1
[1] 0.2222156  # aprx MSE of unbiased variance estimator

v2 = (n-1)*v1/(n+1)      # denom n+1
mean(v2);  mean((v2-1)^2)
[1] 0.8180679  # noticeable bias 
[1] 0.1818552  # smaller MSE

enter image description here

Formal analytic proofs are not difficult.

Addendum per Comment. Although for normal data the usual definition of the sample variance (denominator $n-1$ gives unbiased estimates of population variance $\sigma^2,$ unbiasedness does not survive the nonlinear operation of taking square roots. Thus it is not true that $E(S) = \sigma.$ In particular, for normal samples of size $n=3,$ one has $E(S) \approx 0.866\sigma,$ so that $E(S/0.866) = E(1.155\,S) = \sigma,$

set.seed(1118)
s = replicate(10^6, sd(rnorm(3)))
mean(s)
[1] 0.8863831
k = 1/mean(s)
mean(k*s)
[1] 1
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.