Validity of approximating a covariance matrix by making use of a probability limit? I want to know can we approximate the covariance matrix of a random vector by making use of a probability limit.
Define the linear regression model in matrix form as
$$
\mathbf{Y} = \mathbf{X} \beta + \varepsilon,
$$
where the variance of $\varepsilon$ is $\sigma$.
I am interested in approximating $E[\text{Cov}[A|\mathbf{X}]]$ defined by
$$
E[\text{Cov}[\hat \beta|\mathbf{X}]] = E\bigg[\frac{\sigma^2}{n} \bigg(\frac{\mathbf{X}^T\mathbf{X}}{n}\bigg)^{-1}\bigg] = \frac{\sigma^2}{n} E\bigg[\bigg(\frac{\mathbf{X}^T\mathbf{X}}{n}\bigg)^{-1}\bigg].
$$
The probability limit of $\mathbf{X}^T\mathbf{X}/n$ is
$$
\text{plim}_{n\to \infty} \bigg(\frac{\mathbf{X}^T\mathbf{X}}{n}\bigg) = Q,
$$
where $Q$ is a constant positive definite matrix (see Econometric Analysis by William Greene, eq. 4-19). So the probability limit of the inverse $(\mathbf{X}^T\mathbf{X}/n)^{-1}$ is
$$
\text{plim}_{n\to \infty} \bigg(\frac{\mathbf{X}^T\mathbf{X}}{n}\bigg)^{-1} = Q^{-1}.
$$
For large $n$, I am interested in approximating $E[\text{Cov}[\hat \beta|\mathbf{X}]]$ by using the probability limit, that is, saying something like
$$
E[\text{Cov}[\hat \beta|\mathbf{X}]] \approx \frac{\sigma^2}{n} Q^{-1}, \quad \quad \text{or} \quad \quad E[\text{Cov}[\hat \beta|\mathbf{X}]] \sim \frac{\sigma^2}{n} Q^{-1}.
$$
I have various questions regarding the validity of doing this.
What kind of error are we making if we can do this? Is there a way to account for the error? Is this a situation where we have an approximation that 'holds with high probability'? If we can indeed make this approximation, how do we rigorously state it mathematically (precisely what does $\approx$ or $\sim$ signify)?
 A: In "standard linear regression" with strict exogeneity, $E(\varepsilon \mid \mathbf X) = 0$,
the OP wants to approximate (pursuing a theoretical result) the unconditional variance of $\hat \beta$ by using the probability limit of the the moment matrix.
By the Law of Total Variance and the fact that $E(\hat \beta \mid \mathbf X) = \beta$, we have that the  unconditional variance is
$${\rm V}(\hat \beta) = \sigma^2 \cdot E\Big[(\mathbf X' \mathbf X)^{-1}\Big] = \frac{\sigma^2 }{n}\cdot E\Big[(n^{-1}\mathbf X' \mathbf X)^{-1}\Big]$$
We approximate this by
$${\rm  V}(\hat \beta) \approx \frac{\sigma^2}{n} \cdot Q^{-1},$$
where
$$Q = {\rm plim}\left(n^{-1}\mathbf X' \mathbf X\right) = E(\mathbf x \mathbf x')$$
where $\mathbf x$ is the typical row vector of $\mathbf X$ and is used because at the limit the matrix $X$ has infinite row dimension, so  it  would be inappropriate to use it as the result of a limiting expression.
In words, instead of the expected value of the inverse, we use the inverse of the expected value.
The approximation error is
$$\delta(n) =(\sigma^2 /n) \cdot \Big[E[(n^{-1}\mathbf X'\mathbf X)^{-1}] - [E(\mathbf x'\mathbf x)]^{-1}\Big].$$
We have that $(n^{-1}\mathbf X'\mathbf X)^{-1} \longrightarrow_p [E(\mathbf x'\mathbf x)]^{-1}$, so
$$E[(n^{-1}\mathbf X'\mathbf X)^{-1}] - [E(\mathbf x'\mathbf x)]^{-1} \longrightarrow  E[(E(\mathbf x'\mathbf x))^{-1}] - [E(\mathbf x'\mathbf x)]^{-1} = 0, $$
so this expression is $o(1)$. Also, $(\sigma^2/n) = O(1/n)$. Therefore,
$$\delta(n) = O(1/n)\cdot o(1) = o(1\cdot 1/n) = o(1/n).$$
So the approximation error goes to zero faster than $n$ goes to infinity.
UPDATE
Can we improve on the $o_p(1) / o(1)$ rate of convergence of
$$E[(n^{-1}\mathbf X'\mathbf X)^{-1}] - [E(\mathbf x'\mathbf x)]^{-1},\;\;\; ?$$
Apparently, the OP needs that. Let's see.
The OP mentioned a remark in Bruce Hansen's Econometrics textbook, about the OLS estimator having a faster convergence rate than $o_p(1)$. Hansen derives this after it obtains the rate of scaling needed for the asymptotic distribution. And since this latter is $O_p(n^{-1/2})$, it follows that multiplying the estimator $\hat \beta_n - \beta$ by something larger than unity ($n^0$) but lower than $n^{1/2}$ will not hinge his journey towards zero.
To clear the eye, we are examining the rate of convergence of
$$E(h_n) - c,\;\;\; c\; {\rm =\;constant}, \;\;\; h_n = O_p(1), \;h_n - c \to_p 0.$$
Now, to apply the Hansen approach, we would need to be able to say something about the distribution (if it exists) of
$$n^{\delta} (h_n - c).$$
If we can prove that, for some $\delta >0$ the above converges in distribution, then we can apply the logic of Hansen, and argue that $\exists \, \gamma,  0<\gamma < \delta$ for which
$$n^{\gamma}(h_n - c) \to_p 0$$
and so
$$(h_n - c) = o_p(1/n^{\gamma}) \implies  E(h_n -c) = o(1/n^{\gamma}).$$
