# How does the covariance matrix of the predictors in multiple regression relate to the matrix $(\mathbf{X}^T \mathbf{X})^{-1}$? [duplicate]

Note in advance, due to my question previously being marked as a duplicate, that the question I ask here is concerned with the relationship between $$(\mathbf{X}^T \mathbf{X})^{-1}$$ and $$\text{Cov}[\mathbf{X},\mathbf{X}]$$, not the relationship between $$(\mathbf{X}^T \mathbf{X})^{-1}$$ and $$\text{Cov}[\mathbf{X},\mathbf{y}]$$ which is covered here.

The matrix formulation of multiple regression for $$n$$ observations is $$\mathbf{Y} = \mathbf{X}^T \beta + \varepsilon,$$ where the error $$\varepsilon$$ has finite variance $$\sigma^2$$. Let $$\mathbf{b}$$ be the estimated coefficients found when we solve the multiple regression problem with least squares.

In Theorem 4.3 of the book Econometric Analysis by William H. Greene it says that asymptotically $$\mathbf{b}$$ is distributed as $$\mathbf{b} = \mathcal{N}\bigg(\beta,\frac{\sigma^2}{n}Q^{-1}\bigg),$$ where $$Q$$ is defined in equation 4.19 as \begin{align*} Q := \text{plim}_{n \to \infty} \frac{\mathbf{X}^T \mathbf{X}}{n}. \end{align*} which is a positive definite matrix. Suppose we write the covariance matrix of $$\mathbf{X}$$ as $$\text{Cov}[\mathbf{X},\mathbf{X}] = \begin{bmatrix} \sigma_{X_1 X_1} & \sigma_{X_1 X_2} & \dots \\ \sigma_{X_2 X_1} & \sigma_{X_2 X_2} & \dots \\ \vdots & & \ddots \\ \sigma_{X_n X_1} & \dots & \dots & \sigma_{X_n X_n} \end{bmatrix},$$ where $$\sigma_{X_i X_j} = E[(X_i-E[X_i])(X_j-E[X_j])],$$

I want to know how the matrix $$(\mathbf{X}^T \mathbf{X})^{-1}$$ relates to the covariance matrix $$\text{Cov}[\mathbf{X},\mathbf{X}]$$.

For example, if we modify the elements of $$\mathbf{X}$$ to make them more collinear, the off-diagonal elements of $$\text{Cov}[\mathbf{X},\mathbf{X}]$$ will increase in magnitude and $$(\mathbf{X}^T \mathbf{X})^{-1}$$ will become 'harder' to invert. But I am looking for a precise mathematical expression that relates the two matrices.

Can $$(\mathbf{X}^T \mathbf{X})^{-1}$$ be expressed in terms of $$\text{Cov}[\mathbf{X},\mathbf{X}]$$ somehow? E.g., can the elements of $$(\mathbf{X}^T \mathbf{X})^{-1}$$ be written in terms of the elements of $$\text{Cov}[\mathbf{X},\mathbf{X}]$$?

• I closed your original question as a duplicate because my answer to the duplicate explicitly describes the relationship you are looking for. It is an abuse of this site to repost a closed question: please see our help center for more about how it works. – whuber Dec 1 '20 at 17:20

Maybe I don't understand your question. but if you have centered $$X$$, then $$\Sigma=X^TX/n$$.

Hence, $$(X^TX)^{-1}=n^{-1}\,\Sigma^{-1}$$.

> X <- scale(MASS::Boston[, 1:3], scale = FALSE)

> Cov <- function(X) ((nrow(X) - 1) / nrow(X)) * cov(X)

> Cov(X)
crim        zn     indus
crim   73.84036 -40.13648  23.94492
zn    -40.13648 542.86184 -85.24385
indus  23.94492 -85.24385  46.97143

> (t(X) %*% X) / nrow(X)
crim        zn     indus
crim   73.84036 -40.13648  23.94492
zn    -40.13648 542.86184 -85.24385
indus  23.94492 -85.24385  46.97143

> solve(Cov(X)) / nrow(X)
crim            zn         indus
crim   3.207972e-05 -2.742849e-07 -1.685125e-05
zn    -2.742849e-07  5.093748e-06  9.383968e-06
indus -1.685125e-05  9.383968e-06  6.769460e-05

> solve(t(X) %*% X)
crim            zn         indus
crim   3.207972e-05 -2.742849e-07 -1.685125e-05
zn    -2.742849e-07  5.093748e-06  9.383968e-06
indus -1.685125e-05  9.383968e-06  6.769460e-05

• Right: and the original duplicate explains in detail how the matrices are related when they are not initially centered. – whuber Dec 1 '20 at 17:21
• Sorry. I didn't notice the original duplicate. I'll delete the answer. – Zen Dec 1 '20 at 17:25
• It's not your fault: the OP caused this inconvenience by circumventing the norms of the site. – whuber Dec 1 '20 at 17:28