3
$\begingroup$

Suppose $$Y_i=X_i'\beta+\epsilon_i$$ with $E(\epsilon_i|X_i)=0$ and $E\epsilon^2_i=\sigma^2$ and I estimate $\sigma^2$ using $s^2=\frac{1}{n}\sum_{i=1}^n (Y_i-X_i'\widehat{\beta})^2$, where $\widehat{\beta}$ is the OLS estimator of $\beta$.

What's the asymptotic distribution of (a properly scaled and normalized version of) $s^2$?

$\endgroup$
9
  • 1
    $\begingroup$ I'm pretty sure that $n \times s^2$ is distributed as $\chi^2_{n-k}$ where $k$ is the dimension of $\beta$. But it's not asymptotic in the sense that, under the OLS assumptions, It's true for any $n$. But check say Neter and Wasserman or Draper and Smith to confirm what I said. It's from memory and the memory isn't what it once was. $\endgroup$
    – mlofton
    Commented May 7 at 5:39
  • 1
    $\begingroup$ Note that you forgot to square the $(Y - X \hat{\beta})$ term in the summation. $\endgroup$
    – mlofton
    Commented May 7 at 5:41
  • 1
    $\begingroup$ Since you don't mention normality as an assumption, the distribution of the sample variance is not chi-square. It has an asymptotically normal distribution involving the kurtosis of the conditional Y distribution. $\endgroup$ Commented May 7 at 9:34
  • 1
    $\begingroup$ @mlofton You seem to equate "asymptotic" with "has a limit of," but the two are distinctly different mathematical concepts. See en.wikipedia.org/wiki/Asymptotic_expansion for some discussion of the distinction. $\endgroup$
    – whuber
    Commented May 7 at 18:55
  • 1
    $\begingroup$ @whuber: I found a good document on asymptotic expansions in statistics so I will read that as time permits. The wiki you pointed to was decent but my brain needs more detail and a longer explanation. Thanks for your help. Oh, note that I was claiming that the $\chi^2_{n-k}$ was the exact distribution rather than an asymptotic distribution. But, that's only true when the normality assumption for $\epsilon$ is made. $\endgroup$
    – mlofton
    Commented May 8 at 4:15

2 Answers 2

4
$\begingroup$

This answer extends the sample mean case studied in Asymptotic distribution of $\sqrt{n}(\hat{\sigma}_{1}^{2}-\sigma^2)$ to the linear regression setup.

Let us assume an iid sample from a linear model $y_i=x_i'\beta+\epsilon_i$ with first four conditional error moments $\{E(\epsilon_i^p|x_i)\}_{p=1}^4=(0,\sigma^2,\mu_3,\mu_4)'$. Note $$\varsigma^2:=Var(\epsilon_i^2|x_i) = E(\epsilon_i^4|x_i)-[E(\epsilon_i^2|x_i)]^2=\mu_4-\sigma^4$$ and assume $0<\varsigma^2 <\infty$. That is, we require higher-moment assumptions not required for, e.g., "mere" consistency of the OLS estimator. In addition to assuming homoskedasticity, we also assume what one might call "homokurtosis", i.e., $Var(\epsilon_i^2|x_i)$ does not depend on $x_i$.

We may then plug in the model $y_i=x_i'\beta+\epsilon_i$ to get \begin{align*} s^2 & = \frac{1}{n}\sum_{i=1}^n (y_i-x_i'\hat\beta)^2\\ & = \frac{1}{n}\sum_{i=1}^n (\epsilon_i + (\beta-\hat\beta)'x_i))^2\\ & = \frac{1}{n}\sum_{i=1}^n \epsilon_i^2 + 2 (\beta-\hat\beta)'\frac{1}{n} \sum_{i=1}^nx_i\epsilon_i + \frac{1}{n}\sum_{i=1}^n[(\beta-\hat\beta)'x_i]^2\\ & = \underbrace{\frac{1}{n}\sum_{i=1}^n \epsilon_i^2}_{A} + \underbrace{2 (\beta-\hat\beta)'\frac{1}{n} \sum_{i=1}^nx_i\epsilon_i}_B+ \underbrace{(\beta-\hat\beta)'\frac{1}{n}\sum_{i=1}^nx_ix_i'(\beta-\hat\beta)}_C \end{align*}

$B$ and $C$ are asymptotically negligible under suitable assumptions: The OLS estimator is asymptotically normal, i.e., $$\sqrt{n}(\hat\beta-\beta)\to_dN(0,Avar(\hat\beta))$$ and thus converges at rate $\sqrt{n}$, i.e., $\hat\beta-\beta=O_p(n^{-1/2})$.

Under predeterminedness (implied by the exogeneity condition $E(\epsilon_i|x_i)=0$ you state), a WLLN gives $$ \frac{1}{n} \sum_{i=1}^nx_i\epsilon_i\to_pE(x_i\epsilon_i)=0, $$ i.e., $$\frac{1}{n} \sum_{i=1}^nx_i\epsilon_i=o_p(1)$$ Thus, \begin{align*} B & = O_p(n^{-1/2})o_p(1)=o_p(n^{-1/2}) \end{align*} Similarly, with existing second moments of the regressors we have $$ \frac{1}{n}\sum_{i=1}^nx_ix_i'\to_pE(x_ix_i')<\infty, $$ so that $$ \frac{1}{n}\sum_{i=1}^nx_ix_i'=O_p(1), $$ Then, $$ C=O_p(n^{-1/2})O_p(1)O_p(n^{-1/2})=O_p(n^{-1})=o_p(n^{-1/2}) $$

Hence, we may focus on the key term $A$ now. By assumption we have $0<\varsigma^2 <\infty$ and hence, by an application of the central limit theorem with $E(\epsilon_i^2)=\sigma^2$ we obtain that $$ \begin{eqnarray*} \sqrt{n}(s^2-\sigma^2)&=&\sqrt{n}\left(\frac{1}{n}\sum_{i=1}^n \epsilon_i^2+o_p(n^{-1/2})-\sigma^2\right)\\ &=&\sqrt{n}\left(\frac{1}{n}\sum_{i=1}^n \epsilon_i^2-\sigma^2\right) +o_p(1)\\ &\stackrel{d}{\to}&\mathcal{N}(0,\varsigma^2). \end{eqnarray*} $$

Writing $$ \varsigma^2=\mu_4-\sigma^4, $$ highlights the dependence of the asymptotic distribution on higher moments. E.g., for normally distributed errors, $\mu_4=3\sigma^4$, so that $\varsigma^2=2\sigma^4$.

Two illustrations based on normal and Laplace errors, depending on what you comment in:

n <- 2000

errvar <- function(n){
  # y <- rnorm(n) # i.e. an error variance of sigma^2=1
  y <- VGAM::rlaplace(n) # i.e. error variance=2
  x <- rnorm(n)
  limo <- summary(lm(y~x))
  sighatsq <- limo$sigma^2 # here, we still have the (asymptotically negligible) d.o.f. correction
  # sqrt(n)*(sighatsq - 1) # normal case
  sqrt(n)*(sighatsq - 2)
}

MASS::truehist(replicate(50000, errvar(n)), col="lightblue", nbins=50)
xax <- seq(-15, 15, by=.01)
# lines(xax, dnorm(xax, sd=sqrt(2)), lwd=2, col="darkgreen") # note varepsilon=2, normal case
lines(xax, dnorm(xax, sd=sqrt(20)), lwd=2, col="darkgreen") # note 4th moment 24, https://stats.stackexchange.com/questions/10982/moments-of-laplace-distribution, i.e. varepsilon=24-2^2=20

The picture is for the Laplace case:

enter image description here

$\endgroup$
1
  • $\begingroup$ @ChristophHank Thanks a lot, Christoph! This is very clear and helpful! $\endgroup$ Commented May 8 at 3:42
1
$\begingroup$

$\hat{\sigma}^2 \sim \Gamma(\nu/2, \nu/(2 \sigma^2))$ (using the parameterizaton of the Gamma distribution in terms of shape and rate parameter), where $\nu$ is the degrees of freedom and $\sigma^2$ is the true value of the variance. The reference from which I have this is Schmidli et al. 2017, which cannot possibly be the first place that derived this (probably such common knowledge/found in many regression books that they didn't even think it needed to be cited any longer?). I assume you'd prove it by looking at the moment generating function.

$\endgroup$
8
  • $\begingroup$ I am not sure this can be an asymptotic distribution? $\endgroup$ Commented May 7 at 14:13
  • $\begingroup$ @Christoph Sure it is: "asymptotic" does not demand a limit. Indeed, there are always infinitely many answers to a question of this sort, depending on how precisely one wants to measure the asymptotic behavior. It's simpler, more natural, and better to consider the asymptotic behavior of the ratio $\hat\sigma^2/\sigma^2,$ as stated here, than to consider the difference: although both approaches give valid asymptotics, having the exact distribution for any $n$ is better than any approximation to it! $\endgroup$
    – whuber
    Commented May 7 at 18:53
  • 2
    $\begingroup$ Hmm. Could you sketch what then distinguishes an asymptotic distribution from any other? Not necessarily sample size to infinity, but we send something somewhere, no? And surely an exact distribution is better than an approximate one. But could you elaborate why the ratio would be superior here? And my hunch would finally be that the present result requires a distributional assumption on the errors. $\endgroup$ Commented May 7 at 20:00
  • 3
    $\begingroup$ I am with Christoph, this "$\hat{\sigma}^2 \sim \Gamma(\nu/2, \nu/(2 \sigma^2))$" statement definitely needs to assume i.i.d. $N(0, \sigma^2)$ errors and has nothing to do with "asymptotics". The answerer seems didn't fully understand what OP asked. $\endgroup$
    – Zhanxiong
    Commented May 7 at 20:08
  • $\begingroup$ @Björn Thanks! This looks interesting. Where in the paper did they prove it? I checked the papaer and didn't seem to find any asymptotic results. $\endgroup$ Commented May 8 at 4:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.