2
$\begingroup$

I want to estimate the second moment of a distribution. I know the breakdown of the second moment into the mean-squared and variance: $\mathbb{E}[X^2] = (\mathbb{E}[X])^2 + var(X)$.

When I want to estimate the second moment, however, it seems dubious to say $\widehat{\mathbb{E}[X^2]} = (\widehat{\mathbb{E}[X]})^2 + \widehat{var(X)} = \bar{x}^2 + s^2$.

In particular, why not $\widehat{\mathbb{E}[X^2]} = \bar{x}^2 + \widehat{\sigma}_{MLE}^2?$

Both seem like they can be defended as estimators of the second moment, but it is not at all clear to me that they have the nice properties like maximum likelihood and/or unbiasedness that we often like in our estimators.

What are the typical ways of estimating the second moment?

| cite | improve this question | |
$\endgroup$
  • $\begingroup$ Depends on exactly what you mean by $S^2$ and $\hat\sigma^2_{\mathrm{MLE}}.$ // Also, when you say "I want to estimate...," do you mean to use MME, MLE, or UMVUE? $\endgroup$ – BruceET Jul 3 at 20:59
  • $\begingroup$ @BruceET $S^2=\dfrac{1}{n-1}\sum(x_i-\bar{x})^2$ and $\sigma^2_{MLE}$ uses $n$ in the denominator, instead. I’m open to suggestions on estimators but am curious if there’s a go-to estimator the way that basically everyone uses $S^2$ for variance. $\endgroup$ – Dave Jul 3 at 21:40
  • $\begingroup$ In practice, the unbiased estimator is most commonly used, but a minority statisticians prefer the version with $n$ in the denominator. // In a theoretical discussion you have to read carefully to see whether the MLE or the UMVUE of $\sigma^2$ is discussed. // For normal data, $S^2$ does have larger MSE than the version with denom $n,$ but a version with denom $n+1,$ having even smaller MSE, is not generally used because of its large bias to underestimate $\sigma^2.$ $\endgroup$ – BruceET Jul 3 at 21:58
1
$\begingroup$

Consider $X \sim \mathsf{Norm}(\mu = 5, \sigma=2),$ Then (according to Wikipedia on normal distribution or your text): $$E(X) = 5,\, Var(X) = 2^2 = 4,\, E(X^2) = \mu^2 + \sigma^2 = 25 + 4 =29.$$

The following simulation in R, shows estimates from a million samples of size $n=10$ from this distribution: $$E(\bar X) = E(A) \approx 5,\, E(S^2) \approx 4,\, E(X^2)=E(Q) = E\left(\frac 1n \sum_{i=1}^n X_i^2\right) \approx 29,$$ where $S^2$ is the unbiased estimate of $\sigma^2.$ With a million samples we can expect 2 or 3 significant digits of accuracy.

set.seed(2020)
m = 10^6;  n = 10
x = rnorm(m*n, 5, 2)
DTA = matrix(x, nrow=m)
a = rowMeans(DTA)
q = rowMeans(DTA^2)
mean(a);  mean(s^2);  mean(q)
[1] 4.999994  # aprx E(A) = 5
[1] 3.998543  # aprx Var(X) = 4
[1] 28.99873  # aprx 25 + 4 = 29

If data are from $\mathsf{Exp}(\mu = 5),$ then $$E(X) = \mu = 5,\, SD(X) = \sigma = \mu = 5,\, Var(X) = \mu^2=25,\, E(X^2) = 2\mu^2 = 50.$$ A simulation, similar to the one above for normal data, is as follows:

set.seed(703)
m = 10^6;  n = 10
x = rexp(m*n, .2)
DTA = matrix(x, nrow=m)
a = rowMeans(DTA)
q = rowMeans(DTA^2)
mean(a); mean(q)
[1] 4.998014    # aprx E(X) = 5
[1] 49.96277    # aprx E(X^2) = 50
| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Perhaps I don’t follow your simulation, but it seems like the bias-variance decomposition is a good estimator when the distribution is normal but AWFUL when the distribution is exponential. Am I getting it? (Never mind. I forgot to square the mean. It’s just about dead on.) $\endgroup$ – Dave Jul 3 at 21:50
  • $\begingroup$ I’d made a mistake in reading the code. I need to review it in more detail. $\endgroup$ – Dave Jul 3 at 22:08
  • $\begingroup$ I don't see exactly how the 'awful' part comes from my simulation for exponential data, but I took out unnecessary parts of the 2nd simulation in case they were confusing. // Key fact is that the exponential dist'n has only one parameter. If that parameter is taken to be $\mu$ and $\bar X$ estimates $\mu,$ then it seems foolish to use $S^2$ to estimate $\sigma^2 = \mu^2.$ Certainly, $S^2$ is neither MLE nor UMVUE. $\endgroup$ – BruceET Jul 3 at 22:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.