Let the multiple linear regression model : $\mathbf Y=\mathbf X\mathbf \beta+\mathbf \epsilon,$ where $\epsilon\sim N(\mathbf 0, \sigma^2\mathbf I)$.

Least squares estimates, $ \hat{\mathbf\beta}=(\mathbf X'\mathbf X)^{-1}\mathbf X'\mathbf Y$.

The population variance of the estimator is $$Var(\hat{\mathbf\beta})=\sigma^2(\mathbf X'\mathbf X)^{-1}=\frac{\sigma^2}{n}(\frac{1}{n}\mathbf X'\mathbf X)^{-1}$$.

It is written in a lecture material of a renowned professor (reference: p.16) that:

$Var(\hat{\mathbf\beta})=\frac{\sigma^2}{n}(\frac{1}{n}\mathbf X'\mathbf X)^{-1}$ is $O(\frac{1}{n})$ and the convergence is at the rate of $\frac{1}{\sqrt n}$.

I am not understanding:

(1) How $Var(\hat{\mathbf\beta})=\frac{\sigma^2}{n}(\frac{1}{n}\mathbf X'\mathbf X)^{-1}$ is $O(\frac{1}{n})$? Why is this not $O(1)$, since $n$ cancels out in $Var(\hat{\mathbf\beta})=\frac{\sigma^2}{n}(\frac{1}{n}\mathbf X'\mathbf X)^{-1}$ and only a constant term remains?

(2) How is to calculate the convergence rate, which is here $\frac{1}{\sqrt n}$?

  • $\begingroup$ "It is written in a lecture material of a renowned professor" -> do you have any link? $\endgroup$ Dec 12, 2016 at 3:11
  • 1
    $\begingroup$ @FranckDernoncourt slideplayer.com/slide/3413492 page number is 16. $\endgroup$
    – user 31466
    Dec 12, 2016 at 3:15
  • $\begingroup$ matrix $X$ is $n$ by $k$. $\endgroup$ Dec 12, 2016 at 3:28
  • $\begingroup$ @MatthewGunn yes, $\mathbf X$ is a $n\times k$ matrix. $\endgroup$
    – user 31466
    Dec 12, 2016 at 3:32

2 Answers 2


\begin{align*} \mathrm{Var}\left(\hat{\beta} \right) &= \frac{\sigma^2}{n} \left( \frac{1}{n} X'X \right) ^{-1} \\ &= \frac{\sigma^2}{n} \left( \frac{1}{n} \sum_{i=1}^n \mathbf{x}_i \mathbf{x}_i' \right) ^{-1} \end{align*}

And as $n \rightarrow \infty$ you have $\frac{1}{n}\sum_{i=1}^n \mathbf{x}_i \mathbf{x}_i' $ converging in probability (by Kolmogorov's Law of Large Numbers for iid data) to the constant matrix $A = \mathrm{E}[\mathbf{x} \mathbf{x}']$ where $\mathbf{x}$ is a random vector, an observation drawn from the same distribution as your data.

Loosely, you can think of it as:

$$\mathrm{Var}\left(\hat{\beta} \right) \approx \frac{1}{n}\sigma^2 A^{-1}$$

I think what you may have been missing is that there's $n$ embedded in $X'X$.

Notation notes:

$\mathbf{x}_i$ is a column vector $k$ by 1 denoting the $i$th observation. The data matrix is then an $n$ by $k$ matrix:

$$ X = \begin{bmatrix} \mathbf{x}_1' \\ \mathbf{x}_2' \\ \ldots \\ \mathbf{x}_n' \end{bmatrix}$$

I use bold letters to denote vectors.

  • $\begingroup$ I think you have some typos: $\mathrm{E}[\mathbf{x}_i \mathbf{x}_i']$ is not a particularly meaningful expression in the context in which it is being used here. $\endgroup$
    – cardinal
    Dec 12, 2016 at 3:41
  • $\begingroup$ @cardinal Could you clarify what you mean? This is a commonly used notation: $\mathbf{x}_i$ is a vector denoting the $i$th observation. $\mathbf{x}_i \mathbf{x}_i'$ is the outer product. $\endgroup$ Dec 12, 2016 at 3:47
  • $\begingroup$ That is, as $\left( \frac{1}{n}\mathbf X' \mathbf X\right) ^{-1}$ converges in probability to $\mathrm{E}[\mathbf{x}_i \mathbf{x}_i']$ (or eqivalently, $\mathrm{E}[\mathbf X' \mathbf X]$, isn't it?), so it becomes constant and thus $\mathrm{Var}\left(\hat{\beta} \right)$ is $O(1/n)$. It is now clear. Thank you. But can I extract the result of the convergence rate $1/\sqrt n$ from here? $\endgroup$
    – user 31466
    Dec 12, 2016 at 3:51
  • $\begingroup$ Matthew: $i$ is the index in your original summation. It is, at least, unconventional to have some indeterminate $i$ floating around in a statement regarding a limit. (And, in any case, in this particular setting, the theory would not even require the expectation of any particular $\mathbf x_i$ to exist.) $\endgroup$
    – cardinal
    Dec 12, 2016 at 3:52

If the variance of the estimator $\text{Var}(\hat{\mathbf\beta})=\sigma^2(\mathbf X'\mathbf X)^{-1}$ is $O(1/n)$ (assuming implicitly deterministic regressors or "conditional" on them) then, according to the big-O definition (adjusted for matrices) it must be the case that

$$\lim_{n \to \infty} \left [n\cdot \sigma^2(\mathbf X'\mathbf X)^{-1}\right] $$

exists, it is finite and a non-zero matrix. We have

$$\lim_{n \to \infty} \left [n\cdot \sigma^2(\mathbf X'\mathbf X)^{-1}\right] = \sigma^2 \cdot \lim_{n \to \infty} \left (\frac 1n \mathbf X'\mathbf X\right )^{-1} $$

Part of the standard assumptions of the model is the Grenander conditions that among other things guarantee that the above limit converges to something as we want it to be. So the variance is indeed $O(1/n)$, which also implies that it is not $O(1)$. Multiplying and diving by $n$ does not help to showcase this.

We also have

$$ n\cdot \sigma^2(\mathbf X'\mathbf X)^{-1} = \text{Var}[\sqrt n (\hat \beta-\beta)] $$

and it is from here that the $1/\sqrt n$ "convergence rate" statement comes from. I wonder is it correct since the necessary scaling for the variance is $n$?

  • 1
    $\begingroup$ (+1) In particular, as I tried to softly allude to in the other answer, note that this convergence has nothing to do with any expectation (or law of large numbers or central limit theorem). It is simply a limiting statement about a sequence of fixed matrices (since the standard regression treatment is conditional on $X$). For example, in cases where the experimenter explicitly designs $X$, it is easy for a mad scientist to construct cases where the individual $\mathbf x_i$ have quite pathological behavior as a function of $i$, yet still satisfy the desired limit condition. $\endgroup$
    – cardinal
    Dec 14, 2016 at 22:48
  • $\begingroup$ @cardinal I wonder when your book with all these fascinating pathological cases will come out... :) I hope you will let us know. $\endgroup$ Dec 14, 2016 at 23:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.