# Express correlation matrix of $X$ in terms of $X^{T}X$ (in the OLS context)

In least squares estimation where $Y = \beta X$, how can we find the correlation matrix of $X$ in terms of $X^{T}X$? It seems that $X^{T}X$ is very close in structure to the correlation matrix, but don't know how to get the last step.

• Are you referring to the variance-covariance matrix of the $\hat \beta$ estimators? Can you specify a bit more the step you are referring to? Commented Feb 1, 2016 at 17:21

In case you refer to the variance-covariance matrix of the least squares estimator for $\beta$:

The estimator is

$$\hat{\beta} = (X^TX)^{-1}X^Ty.$$ Substituting $y$ and simplifying yields \begin{align} \hat{\beta} & = \beta + (X^TX)^{-1}X^T \varepsilon \\ \hat{\beta} - \beta & = (X^TX)^{-1}X^T \varepsilon , \end{align} where $\varepsilon$ is a vector of unobserved errors. We can compute the variance of $\hat{\beta}$ as \begin{align} \operatorname{E}[\{\hat{\beta} - \beta\}\{\hat{\beta} - \beta\}^T] & = \operatorname{E}\left[\{(X^TX)^{-1}X^{T} \varepsilon\}\{(X^TX)^{-1}X^T \varepsilon\}^T\right] \\ & = \, (X^T X)^{-1}X^T \operatorname{E}(\varepsilon\varepsilon^T) X (X^T X)^{-1} \\ & = \sigma^2 (X^TX)^{-1}. \end{align} Note that I assume the errors to be homoskedastic and that $X$ is a non-stochastic regressor matrix.

Correlation matrix is the normalized covariance matrix.

The relation between both is:

$$(\operatorname{Corr})_{ij}=\frac{(\operatorname{Var})_{ij}}{\sigma_i \sigma_j}$$

where Corr and Var represent your correlation and variance matrices.

I am expecting you know that the variance-covariance matrix is your $XX^T$.

• Hi, I am wondering how the standard deviations below can be computed for matrices. Wont I need to diagonalize and then take the inverse? Commented Feb 1, 2016 at 19:11
• If the regressors are expressed as deviations from their respective means (and so we exclude a constant term), then the variance-covariance matrix (but not the correlation matrix) is equal to $E(X^TX)$ and the sample VCV is $(1/n)X^TX$. You used the outer product - perhaps you had in mind either that the regressor matrix includes each regressor series in each row (instead of in each column), or the alternative expression $\sum \mathbb x_i\mathbb x_i^T$? Perhaps it would be better to clarify these things in your answer. Commented Feb 1, 2016 at 19:36
• @user1398057, if you don't care for performance, you can diagonalize cov matrix, otherwise, it's better not waste CPU time on calculating the off-diagonal elements. Commented Feb 1, 2016 at 19:54

If your $X$ is a de-meaned standardized design matrix $X_{ij}$, where $i={1,2,\dots,T}$ - observations, and $j={1,2,\dots,N}$ - variables, then the covariance matrix' MLE estimator would be $$\frac{1}{T}X'X$$, where $X'_{ij}=X_{ji}$

If it's not de-meaned and standardized, then the equations don't look as nice: $$\Omega_{ij}=\frac{1}{T}\sum_{t=1}^T \left[ \left(X_{ti}-\mu_i\right)\left(X_{tj}-\mu_j\right) \right]\frac{1}{\sqrt{s^2_is^2_j}}$$ Where $$\mu_j =\frac{1}{T}\sum_{t=1}^T X_{tj}$$ and $$s^2_k=\frac{1}{T}\sum_{t=1}^T\left( X_{tk}-\mu_k\right)^2$$

Now, this is all dull standard stuff from text books, where implicitly assumed $N/T\to 0$. The reality is more interesting when this does not hold. For instance, in finance and machine learning often $N\sim T$, then things get hairy. Take a look at one of the standard approaches to deal with this situation in this paper by Ledoit called shrinkage.