2
$\begingroup$

Assuming that I do not know much about an underlying distribution. How many data points does one needs to estimate the $n$th moment?

Is there a formula for the $n$th moment? Is there also formulas for the confidence interval of these parameters?

If a general formula for $n$th moment is difficult to determine, what do we know about the lower order ones (as far as the number of points needed for estimation) (e.g the first four)?

Any pointer is appreciated.

Improve this question
$\endgroup$
  • 2
    $\begingroup$ It depends on what you are assuming about the distribution: zero are needed if you assume values for the moments and one in all other cases. Likewise, obtaining confidence limits depends on your distributional assumptions. Estimating the moments also depends on what properties you require of the estimator; for instance, there are different estimators of the fourth moment depending on whether you want it to be unbiased. $\endgroup$ – whuber♦ Apr 9 '20 at 18:55
  • $\begingroup$ Assuming that I do not really know the underlying distribution, can we still estimate the number of points needed ? $\endgroup$ – Steve Apr 9 '20 at 20:02
  • 1
    $\begingroup$ One is the minimum. Obviously it's not a very useful minimum, but to go any further you need to develop a sense of how variable that estimate might be. That requires larger samples. For moments higher than the first, it is crucial to narrow the possible distributions as much as possible, because they tend to be extremely variable. This is a much larger subject than might appear--consider, after all, that it covers all the methods of estimating a mean and is closely connected with methods of estimating variances; and that's just the beginning! $\endgroup$ – whuber♦ Apr 9 '20 at 20:53
  • 2
    $\begingroup$ About 25 years ago I think I saw John Tukey cited as saying off the cuff something with roughly this flavour: you need $10^k$ data points to get a half-way decent estimate of the $k$th moment. (Don't quote me on this: all I trust myself to remember was that the rule of thumb was very discouraging.) I've been hoping to rediscover the reference for about 20 years. $\endgroup$ – Nick Cox Apr 10 '20 at 18:11
  • 2
    $\begingroup$ Either after of before deciding on statistical structure to characterize the quality of an estimator, you'll also need to justify how good it needs to be for your application. To think that there's an answer out there irrespective of the subject matter and objective is common but naive. Should one use the same sample size rule for estimation issues associated with brain surgery, baking bread, testing for antibodies, etc. ? $\endgroup$ – JimB May 17 '20 at 19:56
6
$\begingroup$

The raw moments $\mathbb E[X^k]$ all admit unbiased estimators based on an iid sample of size $n$ from the distribution behind the expectation, as for instance$$\frac{1}{n} \sum_{i=1}^n X_i^k$$

As demonstrated in the final page reproduced below Paul Halmos wrote a now famous paper in 1946, called the theory of unbiased estimation, where he gives necessary and sufficient conditions for the existence of unbiased estimators of some expectation based on an iid sample of size $n$ from the distribution behind the expectation.

In particular, he studies the existence of unbiased estimators of the $k$-th centred or central moments$$\mu_k=\mathbb E[(X-\mathbb E\{X\})^k$$for which he shows

  1. that they only exist when $k\le n$
  2. that they can only be expressed as a rescaling of the empirical moment$$\hat\mu_k^n = \frac{1}{n}\sum_{i=1}^n (X_i-\bar{X}_n)^k= \frac{1}{n}\sum_{i=1}^n (X_i-\hat\mu_1^n)^k$$when $k\le 3$. For larger values of $k\le n$, the unbiased estimator of $\mu_k$ also depends on $\mu_\ell^n$ for $1\le\ell\le k-1$.

Note however that $\hat\mu_k^n$ is a converging estimator of $\mu_k$ (as $n$ grows to infinity).

enter image description here

Note however that it is always possible to turn biased estimators into unbiased estimators if sequential sampling is available. Using coupling and stopping time and a telescoping sum argument, as demonstrated by Glynn and Rhee (Exact estimation for Markov chain equilibrium expectations.Journal of Applied Probability, 51(A):377–389, 2014.)

Improve this answer
$\endgroup$
  • 1
    $\begingroup$ In the first item in the enumeration, should "only exist when $k \leq m$" be "... $k \leq n$"? $\endgroup$ – jochen May 16 '20 at 9:19
1
$\begingroup$

General Formula

The general formula for the $n\text{th}$ moment for a function $f(x)$ (from https://en.wikipedia.org/wiki/Moment_(mathematics)) is

$$ \mu_n = \int_{-\infty}^\infty (x - c)^n f(x) dx $$

where $c = 0$ if you're calculating the first moment (the mean), and $c = $ the mean otherwise.

With $k$ data points, this can be estimated as

$$ \hat \mu_n = E[ (x - c)^n ] = \frac{\sum_i^k(x_i - c)^n}{k} $$

For statistical purposes, you'll want to look into the Standardised moments

How much data

At minimum, you need $n$ data points to have an estimate the $n$th moment. You need one to estimate the mean, 2 for the variance, 3 for skew, and so on. Obviously, these will be extremely poor estimates, since these are the absolute minima.

Confidence Intervals

Calculating confidence intervals for higher-order moments in general is tricky. Wright and Herrington (2011) provide a way of estimating them using bootstrap samples.

Improve this answer
$\endgroup$
  • 2
    $\begingroup$ I do not understand why one needs $k$ observations for the $k$-th moment since $X_1^k$ is an unbiased estimator of $\mathbb{E}[X^k]$. $\endgroup$ – Xi'an Apr 10 '20 at 14:32
  • $\begingroup$ That's a fair point. My thinking is that these are the minima needed for these values to be meaningful. You can't sensibly talk about the variance of 1 value, skew of 2 values, or kurtosis of 3 values, I think? $\endgroup$ – Eoin Apr 10 '20 at 15:06
  • $\begingroup$ What if we want to obtain a "good" estimate of the moments--not just a poor one? $\endgroup$ – Steve Apr 10 '20 at 15:20
  • $\begingroup$ I believe this depends on the distribution of your data. You need to decide how small you need the standard error to be to count as "good", and use the bootstrap (citation in my answer) to calculate the SE for the data you have. $\endgroup$ – Eoin Apr 10 '20 at 15:23
  • 2
    $\begingroup$ Dividing by $k$ returns an unbiased estimator. This is not necessarily a "good" property but this is one usually sought in statistical estimation. The denominator of the variance estimator is $k-1$ because of this unbiased property, but the variance is not the second moment, being centred first. $\endgroup$ – Xi'an Apr 10 '20 at 16:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.