0
$\begingroup$

I have often read that thanks to the CLT, the residuals of a model are asymptotically normal. This argument always seemed odd to me since CLT states that

The sum of a number of independent and identically distributed random variables are asymptotically distributed.

Since a residual is generally a difference and not a sum of iid random variables, how can we can claim that the CLT applies?

$\endgroup$
5
  • 1
    $\begingroup$ can you give an example where someone says this? It might help to have context of what was actually said $\endgroup$
    – Glen_b
    Jun 2, 2018 at 7:09
  • $\begingroup$ Here quora.com/Why-is-the-error-term-normally-distributed at the end of the first response and here www3.kau.se/kurstorg/files/k/C10B99481638416B58RqFF5CC0CE/…, page 8 for instance. $\endgroup$
    – mobupu
    Jun 2, 2018 at 10:59
  • $\begingroup$ Unfortunately, neither of those is actually making the claim you asked about. Your first link is about the normality of the error term, not the residuals; the nearest it comes is at one point it uses the appearance of residuals as an argument for making the assumption about the error term. The second calls "u" the residual on p8 but has previously and unambiguously defined it as the error term (see p2 and p3), so the reference to it being the residual on p8 is apparently a mistake; indeed the second dot-point on p8 makes it clear that it's still talking about $u$ as an error term. $\endgroup$
    – Glen_b
    Jun 2, 2018 at 12:10
  • $\begingroup$ Errors are not residuals but in practice, it is our only estimate of errors, I should have added that I was speaking of residuals as estimation of errors, granted. $\endgroup$
    – mobupu
    Jun 2, 2018 at 13:11
  • 1
    $\begingroup$ It is now unclear exactly what you're asking -- are you now asking why errors might be assumed normal, or a claim that the CLT implies that the residuals you use to estimate the errors should be normal? (And if the second, under what circumstances?) $\endgroup$
    – Glen_b
    Jun 3, 2018 at 0:59

3 Answers 3

3
$\begingroup$

What i think you are referring to is that estimators are often asymptotically normal with increase of the sample size. It's a mathematically correct statement that follows from more general formulation of CLT, Lyapunov's CLT. It doesn't require having identical distribution of variables, just independence. For example, OLS estimator is $\hat{\beta}=\beta+P_{x}\epsilon$, where projection operator $P_{x}=(X^{T}X)^{-1}X^{T}$, vector of errors is $\epsilon$. In other words the difference from the actual $\beta$ is linear combination of errors, which are independently distributed. Hence by Lyapunov CLT this difference is asympyotically normally distributed.

In terms of what you literally wrote, i.e. "residuals of a model are asymptotically normal" it is not clear what asymptotic limit we are talking about here. There is a sense in which errors themselves could be approximately normal, i.e. when they are modeled as a sum of very large number of unknown factors. When this assumption is applicable, once again Lyapunov CLT suggests they are approximately normal. Again, i am not sure that is what you are asking, because strictly there is no asymptotical equality here. Also, i suspect there is another confused notion about squared residuals being distributed normally. It would be more optimal to ask the question more specifically.

$\endgroup$
4
$\begingroup$

You are right to be skeptical - The central limit theorem (CLT) says nothing about the distribution of the residuals in a statistical model.$^\dagger$ They might be normal, or not, depending on the underlying distribution of the error terms. The CLT is a theorem that concerns the asymptotic distribution of the standardised sample mean, and other standardised quantities formed by summation of variables. It is unfortunately misinterpreted in such an expansive way that it is attributed to all sorts of quantities it does not apply to. Misinterpretation of the CLT as an assertion of asymptotic normality of the underlying values used in model is unfortunately a ubiquitous error.


$^\dagger$ The one exception to this would be if the error terms in the model are themselves formed as a sum of lots of other IID quantities, etc., such that the CLT would apply to the distribution of the error terms. Although theoretically possible, this is not how statistical models are generally framed.

$\endgroup$
3
$\begingroup$

the residuals of a model are asymptotically normal.

The residuals cannot be asymptotically normal. The phrasing doesn't make a sense. What can be asymptotically normal is some function of those residuals such as the coefficient estimate, so that when the sample size increases the function starts to act like a normal distributed variable.

When you increase the sample size residuals do not suddenly turn into normal. It would be like saying that under CLT in a large sample i.i.d. random variables become normally distributed, they do not! A properly chosen linear combination of random variables will become normal under the conditions of CLT, but not the random variables themselves

$\endgroup$
1
  • $\begingroup$ thank you (I made a typo for squared difference, it is corrected) $\endgroup$
    – mobupu
    Jun 2, 2018 at 11:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.