Rate of convergence of $\hat Q_{xx}^{-1} = \left(\frac{\mathbf{X}^T \mathbf{X}}{n}\right)^{-1}$ to the probability limit?

Question

Consider the simple linear regression model. $$ y_i = \beta_0 + \beta_1 x_i + \varepsilon_i, \quad \quad \quad \quad i = 1,2,\dots,n. $$ Let $\mu_x$ and $\sigma_x^2$ represent the mean and variance of the i.i.d. observations $x_1,x_2,\dots,x_n$. The simple linear regression model can be written in matrix form as $$ y = \mathbf{X} \beta + \varepsilon, $$ where $$ \mathbf{X} = \begin{bmatrix} \mathbf{x_1}^T \\ \mathbf{x_2}^T \\ \vdots \\ \mathbf{x_n}^T \end{bmatrix} = \begin{bmatrix} 1 & x_1 \\ 1 & x_2 \\ \vdots & \vdots \\ 1 & x_n \end{bmatrix}. $$

Regarding the large sample theory for regression, it is known that $$ \bigg(\frac{\mathbf{X}^T \mathbf{X}}{n}\bigg) \stackrel{p}{\longrightarrow} E[\mathbf{x}_i \mathbf{x}_i']. $$ This is shown in equation (6.1) in these notes, where the author uses the notation \begin{align} \hat Q_{xx} = \bigg(\frac{\mathbf{X}^T \mathbf{X}}{n}\bigg), \\ Q_{xx} = E[\mathbf{x}_i \mathbf{x}_i']. \end{align}

Sticking with the notation in those notes, we then have $$ \hat Q_{xx}^{-1} \stackrel{p}{\longrightarrow} Q_{xx}^{-1}. $$ I want to find the actual rate of convergence in probability of $\hat Q_{xx}^{-1}$ to $Q_{xx}^{-1}$. The above expression means that $\hat Q_{xx}^{-1} - Q_{xx}^{-1} = o_p(1)$, but I am hoping that the convergence can be shown more precisely or by shown to be faster, possibly something like $\hat Q_{xx}^{-1} - Q_{xx}^{-1} = O_p(n^{-1/2})$ or $o_p(n^{-1/2})$.

Since we are considering simple linear regression we have an explicit representation for $\hat Q_{xx}^{-1}$: $$ \begin{align} \hat Q_{xx}^{-1} = n (\mathbf{X}^T \mathbf{X})^{-1} & = \frac{1}{\sum_{i=1}^n(x_i - \overline x)^2} \begin{bmatrix} \sum_{i=1}^n x_i^2 & - \sum_{i=1}^n x_i \\ - \sum_{i=1}^n x_i & n \end{bmatrix}, \\ & = \frac{1}{\frac{1}{n}\sum_{i=1}^n(x_i - \overline x)^2} \begin{bmatrix} \frac{1}{n} \sum_{i=1}^n x_i^2 & - \frac{1}{n} \sum_{i=1}^n x_i \\ - \frac{1}{n} \sum_{i=1}^n x_i & \frac{1}{n} n \end{bmatrix}, \\ & = \frac{1}{s^2} \begin{bmatrix} \gamma & - \overline x \\ - \overline x & 1 \end{bmatrix}. \end{align} $$

Then we have:

1. $\sqrt{n}(s^2 - \sigma_x^2) \stackrel{d}{\longrightarrow} \mathcal{N}(0,E[x_i - \mu_x^4])$ (shown here)
2. $\sqrt{n}(\overline x - \mu_x) \stackrel{d}{\longrightarrow} \mathcal{N}(0,\sigma_x^2)$
3. $\sqrt{n}(\gamma - (?)) \stackrel{d}{\longrightarrow} \mathcal{N}(0,(?))$ (I'm not sure about this one)

In any, case the numerator and denominator of each element of the $2\times 2$ matrix $\hat Q_{xx}^{-1}$ is converging in distribution, and since they are converging to a constant, we get convergence in probability. Due to the $\sqrt{n}$ term at the front of each expression it appears we can say that the convergence of $\hat Q_{xx}^{-1}$ to $Q_{xx}^{-1}$ really is faster than $o_p(1)$. It appears we can say $$ \hat Q_{xx}^{-1} - Q_{xx}^{-1} = O_p(n^{-1/2}). $$

Is this in fact the case, do we really have a rate of convergence of $O_p(n^{-1/2})$ or have I made a mistake?

Answer 1

It would be easier to use block-matrix notation, but it's ok.

Under the implied regularity assumptions that you have an i.i.d. sample, hence ergodic, and that the various expected values exist, your "$\gamma$", let's denote it $\gamma_n$, converges to $E(x^2)$.

You obtained

$$\hat Q_{xx}^{-1} - Q_{xx}^{-1} = \frac{1}{s^2} \begin{bmatrix} \gamma_n & - \overline x \\ - \overline x & 1 \end{bmatrix} \;-\;\frac{1}{\sigma^2_x} \begin{bmatrix} E(x^2) & - \mu_x\\ - \mu_x & 1 \end{bmatrix}$$

$$\implies \sqrt{n}\left(\hat Q_{xx}^{-1} - Q_{xx}^{-1}\right) = \begin{bmatrix} \sqrt{n}\left(\frac{\gamma_n}{s^2}-\frac{E(x^2)}{\sigma^2_x}\right) & -\sqrt{n}\left(\frac{\overline x}{s^2}-\frac{\mu_x}{\sigma^2_x}\right) \\ -\sqrt{n}\left(\frac{\overline x}{s^2}-\frac{\mu_x}{\sigma^2_x}\right) & \sqrt{n}\left(\frac{1}{s^2}-\frac{1}{\sigma^2_x}\right) \end{bmatrix}$$

Now you have to show that each element of this last matrix converges in distribution to a random variable, or to a constant. In such a case you will get that

$$\hat Q_{xx}^{-1} - Q_{xx}^{-1} = O_p(n^{-1/2}),$$

and so that it is $o_p(n^{-\delta}),\; 0 \leq \delta < 1/2$, and therefore that convergence is faster than just $o_p(1)$.

For the upper row (and so the left lower element), this is true using the i.i.d. sample assumption, Slutsky's Lemma and the symmetry of the Normal distribution around its mean (they obey a classical CLT). A little care with the lower right element.

See this post, https://stats.stackexchange.com/a/379971/28746 for when we can go from convergence in distribution to convergence in probability (in fact to the even stronger convergence in quadratic mean).