Convergence of $Var(\hat{\mathbf\beta})$ in multiple linear regression model

Question

Let the multiple linear regression model : $\mathbf Y=\mathbf X\mathbf \beta+\mathbf \epsilon,$ where $\epsilon\sim N(\mathbf 0, \sigma^2\mathbf I)$.

Least squares estimates, $ \hat{\mathbf\beta}=(\mathbf X'\mathbf X)^{-1}\mathbf X'\mathbf Y$.

The population variance of the estimator is $$Var(\hat{\mathbf\beta})=\sigma^2(\mathbf X'\mathbf X)^{-1}=\frac{\sigma^2}{n}(\frac{1}{n}\mathbf X'\mathbf X)^{-1}$$.

It is written in a lecture material of a renowned professor (reference: p.16) that:

$Var(\hat{\mathbf\beta})=\frac{\sigma^2}{n}(\frac{1}{n}\mathbf X'\mathbf X)^{-1}$ is $O(\frac{1}{n})$ and the convergence is at the rate of $\frac{1}{\sqrt n}$.

I am not understanding:

(1) How $Var(\hat{\mathbf\beta})=\frac{\sigma^2}{n}(\frac{1}{n}\mathbf X'\mathbf X)^{-1}$ is $O(\frac{1}{n})$? Why is this not $O(1)$, since $n$ cancels out in $Var(\hat{\mathbf\beta})=\frac{\sigma^2}{n}(\frac{1}{n}\mathbf X'\mathbf X)^{-1}$ and only a constant term remains?

(2) How is to calculate the convergence rate, which is here $\frac{1}{\sqrt n}$?

Answer 1

\begin{align*} \mathrm{Var}\left(\hat{\beta} \right) &= \frac{\sigma^2}{n} \left( \frac{1}{n} X'X \right) ^{-1} \\ &= \frac{\sigma^2}{n} \left( \frac{1}{n} \sum_{i=1}^n \mathbf{x}_i \mathbf{x}_i' \right) ^{-1} \end{align*}

And as $n \rightarrow \infty$ you have $\frac{1}{n}\sum_{i=1}^n \mathbf{x}_i \mathbf{x}_i' $ converging in probability (by Kolmogorov's Law of Large Numbers for iid data) to the constant matrix $A = \mathrm{E}[\mathbf{x} \mathbf{x}']$ where $\mathbf{x}$ is a random vector, an observation drawn from the same distribution as your data.

Loosely, you can think of it as:

$$\mathrm{Var}\left(\hat{\beta} \right) \approx \frac{1}{n}\sigma^2 A^{-1}$$

I think what you may have been missing is that there's $n$ embedded in $X'X$.

---

Notation notes:

$\mathbf{x}_i$ is a column vector $k$ by 1 denoting the $i$th observation. The data matrix is then an $n$ by $k$ matrix:

$$ X = \begin{bmatrix} \mathbf{x}_1' \\ \mathbf{x}_2' \\ \ldots \\ \mathbf{x}_n' \end{bmatrix}$$

I use bold letters to denote vectors.

Answer 2

If the variance of the estimator $\text{Var}(\hat{\mathbf\beta})=\sigma^2(\mathbf X'\mathbf X)^{-1}$ is $O(1/n)$ (assuming implicitly deterministic regressors or "conditional" on them) then, according to the big-O definition (adjusted for matrices) it must be the case that

$$\lim_{n \to \infty} \left [n\cdot \sigma^2(\mathbf X'\mathbf X)^{-1}\right] $$

exists, it is finite and a non-zero matrix. We have

$$\lim_{n \to \infty} \left [n\cdot \sigma^2(\mathbf X'\mathbf X)^{-1}\right] = \sigma^2 \cdot \lim_{n \to \infty} \left (\frac 1n \mathbf X'\mathbf X\right )^{-1} $$

Part of the standard assumptions of the model is the Grenander conditions that among other things guarantee that the above limit converges to something as we want it to be. So the variance is indeed $O(1/n)$, which also implies that it is not $O(1)$. Multiplying and diving by $n$ does not help to showcase this.

We also have

$$ n\cdot \sigma^2(\mathbf X'\mathbf X)^{-1} = \text{Var}[\sqrt n (\hat \beta-\beta)] $$

and it is from here that the $1/\sqrt n$ "convergence rate" statement comes from. I wonder is it correct since the necessary scaling for the variance is $n$?