-2
$\begingroup$

Note in advance, due to my question previously being marked as a duplicate, that the question I ask here is concerned with the relationship between $(\mathbf{X}^T \mathbf{X})^{-1}$ and $\text{Cov}[\mathbf{X},\mathbf{X}]$, not the relationship between $(\mathbf{X}^T \mathbf{X})^{-1}$ and $\text{Cov}[\mathbf{X},\mathbf{y}]$ which is covered here.

The matrix formulation of multiple regression for $n$ observations is $$ \mathbf{Y} = \mathbf{X}^T \beta + \varepsilon, $$ where the error $\varepsilon$ has finite variance $\sigma^2$. Let $\mathbf{b}$ be the estimated coefficients found when we solve the multiple regression problem with least squares.

In Theorem 4.3 of the book Econometric Analysis by William H. Greene it says that asymptotically $\mathbf{b}$ is distributed as $$ \mathbf{b} = \mathcal{N}\bigg(\beta,\frac{\sigma^2}{n}Q^{-1}\bigg), $$ where $Q$ is defined in equation 4.19 as \begin{align*} Q := \text{plim}_{n \to \infty} \frac{\mathbf{X}^T \mathbf{X}}{n}. \end{align*} which is a positive definite matrix. Suppose we write the covariance matrix of $\mathbf{X}$ as $$ \text{Cov}[\mathbf{X},\mathbf{X}] = \begin{bmatrix} \sigma_{X_1 X_1} & \sigma_{X_1 X_2} & \dots \\ \sigma_{X_2 X_1} & \sigma_{X_2 X_2} & \dots \\ \vdots & & \ddots \\ \sigma_{X_n X_1} & \dots & \dots & \sigma_{X_n X_n} \end{bmatrix}, $$ where $$ \sigma_{X_i X_j} = E[(X_i-E[X_i])(X_j-E[X_j])], $$

I want to know how the matrix $(\mathbf{X}^T \mathbf{X})^{-1}$ relates to the covariance matrix $\text{Cov}[\mathbf{X},\mathbf{X}]$.

For example, if we modify the elements of $\mathbf{X}$ to make them more collinear, the off-diagonal elements of $\text{Cov}[\mathbf{X},\mathbf{X}]$ will increase in magnitude and $(\mathbf{X}^T \mathbf{X})^{-1}$ will become 'harder' to invert. But I am looking for a precise mathematical expression that relates the two matrices.

Can $(\mathbf{X}^T \mathbf{X})^{-1}$ be expressed in terms of $\text{Cov}[\mathbf{X},\mathbf{X}]$ somehow? E.g., can the elements of $(\mathbf{X}^T \mathbf{X})^{-1}$ be written in terms of the elements of $\text{Cov}[\mathbf{X},\mathbf{X}]$?

$\endgroup$
1
  • $\begingroup$ I closed your original question as a duplicate because my answer to the duplicate explicitly describes the relationship you are looking for. It is an abuse of this site to repost a closed question: please see our help center for more about how it works. $\endgroup$ – whuber Dec 1 '20 at 17:20
1
$\begingroup$

Maybe I don't understand your question. but if you have centered $X$, then $\Sigma=X^TX/n$.

Hence, $(X^TX)^{-1}=n^{-1}\,\Sigma^{-1}$.

> X <- scale(MASS::Boston[, 1:3], scale = FALSE)

> Cov <- function(X) ((nrow(X) - 1) / nrow(X)) * cov(X) 

> Cov(X)
           crim        zn     indus
crim   73.84036 -40.13648  23.94492
zn    -40.13648 542.86184 -85.24385
indus  23.94492 -85.24385  46.97143

> (t(X) %*% X) / nrow(X)
           crim        zn     indus
crim   73.84036 -40.13648  23.94492
zn    -40.13648 542.86184 -85.24385
indus  23.94492 -85.24385  46.97143

> solve(Cov(X)) / nrow(X)
               crim            zn         indus
crim   3.207972e-05 -2.742849e-07 -1.685125e-05
zn    -2.742849e-07  5.093748e-06  9.383968e-06
indus -1.685125e-05  9.383968e-06  6.769460e-05

> solve(t(X) %*% X)
               crim            zn         indus
crim   3.207972e-05 -2.742849e-07 -1.685125e-05
zn    -2.742849e-07  5.093748e-06  9.383968e-06
indus -1.685125e-05  9.383968e-06  6.769460e-05
$\endgroup$
3
  • $\begingroup$ Right: and the original duplicate explains in detail how the matrices are related when they are not initially centered. $\endgroup$ – whuber Dec 1 '20 at 17:21
  • $\begingroup$ Sorry. I didn't notice the original duplicate. I'll delete the answer. $\endgroup$ – Zen Dec 1 '20 at 17:25
  • $\begingroup$ It's not your fault: the OP caused this inconvenience by circumventing the norms of the site. $\endgroup$ – whuber Dec 1 '20 at 17:28

Not the answer you're looking for? Browse other questions tagged or ask your own question.