14
$\begingroup$

Most asymptotic results in statistics prove that as $n \rightarrow \infty$ an estimator (such as the MLE) converges to a normal distribution based on a second-order taylor expansion of the likelihood function. I believe there's a similar result in Bayesian literature, the "Bayesian Central Limit Theorem", which shows that the posterior converges asymptotically to a normal as $n \rightarrow \infty$

My question is - does the distribution converge to something "before" it becomes normal, based on the third term in the Taylor series? Or is this not possible to do in general?

Improve this question
$\endgroup$
2
  • $\begingroup$ (+1) .. good question. The Bayesian Central Limit Theorem is called the Laplace Approximation ie the posterior behaves "more or less" like a normal distribution. (formally posterior converges in distribution to a normal distribution) $\endgroup$
    – suncoolsu
    Nov 4, 2010 at 6:17
  • $\begingroup$ Related: stats.stackexchange.com/questions/191492/… $\endgroup$
    – kjetil b halvorsen ♦
    Mar 16, 2017 at 15:18

4 Answers 4

Reset to default
7
$\begingroup$

You are searching for the Edgeworth series aren't you?

See the Wikipedia article on same.

(note that Edgeworth died in 1926, should be in most famous statisticians? )

Improve this answer
$\endgroup$
5
$\begingroup$

It is not possible for a sequence to "converge" to one thing and then to another. The higher-order terms in an asymptotic expansion will go to zero. What they tell you is how close to zero they are for any given value of $n$.

For the Central Limit Theorem (as an example) the appropriate expansion is that of the logarithm of the characteristic function: the cumulant generating function (cgf). Standardization of the distributions fixes the zeroth, first, and second terms of the cgf. The remaining terms, whose coefficients are the cumulants, depend on $n$ in an orderly way. The standardization that occurs in the CLT (dividing the sum of $n$ random variables by something proportional to $n^{1/2}$--without which convergence will not occur) causes the $m^\text{th}$ cumulant--which after all depends on $m^\text{th}$ moments--to be divided by $(n^{1/2})^m = n^{m/2}$, but at the same time because we are summing $n$ terms, the net result is that the $m^\text{th}$ order term is proportional to $n/n^{m/2} = n^{-(m-2)/2}$. Thus the third cumulant of the standardized sum is proportional to $1/n^{1/2}$, the fourth cumulant is proportional to $1/n$, and so on. These are the higher-order terms. (For details, see this paper of Yuval Filmus for example.)

In general, a high negative power of $n$ is much smaller than a low negative power. We can always be assured of this by taking a sufficiently large value of $n$. Thus, for really large $n$ we can neglect all negative powers of $n$: they converge to zero. Along the way to convergence, departures from the ultimate limit are measured with increasing accuracy by the additional terms: the $1/n^{1/2}$ term is an initial "correction," or departure from the limiting value; the next $1/n$ term is a smaller, more quickly-vanishing correction added to that, and so on. In brief, the additional terms give you a picture of how quickly the sequence converges to its limit.

These additional terms can help us make corrections for finite (usually small) values of $n$. They show up all the time in this regard, such as Chen's modification of the t-test, which exploits the third-order ($1/n^{1/2}$) term.

Improve this answer
$\endgroup$
2
  • $\begingroup$ for some reason, I don't find your answer entirely convincing. I do agree that the distribution needs to be "stretched", and that it's not correct to say it converges to X before it converges to a normal. That would be a mistake on my part. Still I think there should exist some way to scale the distribution such that only the fourth order and above "moments" go towards zero. I need to think a little harder as to what exactly that scaling factor thing would look like, if such a thing were to exist $\endgroup$
    – gabgoh
    Nov 5, 2010 at 6:27
  • 2
    $\begingroup$ @gabgoh I would like to hear more about which aspect(s) of the answer are weak. As far as scaling goes, you're stuck: you have already used up that possibility in standardizing the elements of the sequence. If (hypothetically) some form of scaling would keep the third moments from going to zero, then you would contradict the CLT because the limiting distribution would not be Normal. There's a related issue with asymptotics of estimators. Often you can adjust an estimator to kill higher moments asymptotically (e.g., with bootstrapping): but this still cannot be done by scaling alone. $\endgroup$
    – whuber ♦
    Nov 7, 2010 at 16:54
3
$\begingroup$

Here is an attempt to answer your insightful question. I have seen the inclusion of the 3rd term of the Taylor series to increase the speed of convergence of the series to the true distribution. However, I haven't seen (in my limited experience) the usage of third and higher moments.

As pointed out by John D. Cook in his blogs (here and here), there hasn't been much work done in this direction, apart from the Berry-Esseen theorem. My guess would be (from the observation in the blog about the approximation error being bounded by $n^{1/2}$), as the asymptotic normality of mle is guaranteed at a rate of converge of $n^{1/2}$ ($n$, being sample size), considering higher moments won't improve on the normality result.

Therefore, I guess, the answer to your question should be no. The asymptotic distribution converges to a normal dist.(by CLT, under regularity conditions of Lindberg's CLT). However, using higher order terms may increase the rate of convergence to the asymptotic distribution.

Improve this answer
$\endgroup$
3
$\begingroup$

Definitely not my area, but I'm pretty sure third- and higher-order asymptotics exist. Is this any help?

Robert L. Strawderman. Higher-Order Asymptotic Approximation: Laplace, Saddlepoint, and Related Methods Journal of the American Statistical Association Vol. 95, No. 452 (Dec., 2000), pp. 1358-1364

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.