In least squares estimation where $Y = \beta X$, how can we find the correlation matrix of $X$ in terms of $X^{T}X$? It seems that $X^{T}X$ is very close in structure to the correlation matrix, but don't know how to get the last step.
$\begingroup$
$\endgroup$
1
-
1$\begingroup$ Are you referring to the variance-covariance matrix of the $\hat \beta$ estimators? Can you specify a bit more the step you are referring to? $\endgroup$– Antoni ParelladaCommented Feb 1, 2016 at 17:21
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
In case you refer to the variance-covariance matrix of the least squares estimator for $\beta$:
The estimator is
$$ \hat{\beta} = (X^TX)^{-1}X^Ty. $$ Substituting $y$ and simplifying yields \begin{align} \hat{\beta} & = \beta + (X^TX)^{-1}X^T \varepsilon \\ \hat{\beta} - \beta & = (X^TX)^{-1}X^T \varepsilon , \end{align} where $\varepsilon$ is a vector of unobserved errors. We can compute the variance of $\hat{\beta}$ as \begin{align} \operatorname{E}[\{\hat{\beta} - \beta\}\{\hat{\beta} - \beta\}^T] & = \operatorname{E}\left[\{(X^TX)^{-1}X^{T} \varepsilon\}\{(X^TX)^{-1}X^T \varepsilon\}^T\right] \\ & = \, (X^T X)^{-1}X^T \operatorname{E}(\varepsilon\varepsilon^T) X (X^T X)^{-1} \\ & = \sigma^2 (X^TX)^{-1}. \end{align} Note that I assume the errors to be homoskedastic and that $X$ is a non-stochastic regressor matrix.
$\begingroup$
$\endgroup$
3
Correlation matrix is the normalized covariance matrix.
The relation between both is:
$$(\operatorname{Corr})_{ij}=\frac{(\operatorname{Var})_{ij}}{\sigma_i \sigma_j} $$
where Corr and Var represent your correlation and variance matrices.
I am expecting you know that the variance-covariance matrix is your $XX^T$.
-
$\begingroup$ Hi, I am wondering how the standard deviations below can be computed for matrices. Wont I need to diagonalize and then take the inverse? $\endgroup$ Commented Feb 1, 2016 at 19:11
-
2$\begingroup$ If the regressors are expressed as deviations from their respective means (and so we exclude a constant term), then the variance-covariance matrix (but not the correlation matrix) is equal to $E(X^TX)$ and the sample VCV is $(1/n)X^TX$. You used the outer product - perhaps you had in mind either that the regressor matrix includes each regressor series in each row (instead of in each column), or the alternative expression $\sum \mathbb x_i\mathbb x_i^T$? Perhaps it would be better to clarify these things in your answer. $\endgroup$ Commented Feb 1, 2016 at 19:36
-
$\begingroup$ @user1398057, if you don't care for performance, you can diagonalize cov matrix, otherwise, it's better not waste CPU time on calculating the off-diagonal elements. $\endgroup$– AksakalCommented Feb 1, 2016 at 19:54
$\begingroup$
$\endgroup$
If your $X$ is a de-meaned standardized design matrix $X_{ij}$, where $i={1,2,\dots,T}$ - observations, and $j={1,2,\dots,N}$ - variables, then the covariance matrix' MLE estimator would be $$\frac{1}{T}X'X$$, where $X'_{ij}=X_{ji}$
If it's not de-meaned and standardized, then the equations don't look as nice: $$\Omega_{ij}=\frac{1}{T}\sum_{t=1}^T \left[ \left(X_{ti}-\mu_i\right)\left(X_{tj}-\mu_j\right) \right]\frac{1}{\sqrt{s^2_is^2_j}} $$ Where $$\mu_j =\frac{1}{T}\sum_{t=1}^T X_{tj}$$ and $$s^2_k=\frac{1}{T}\sum_{t=1}^T\left( X_{tk}-\mu_k\right)^2$$
Now, this is all dull standard stuff from text books, where implicitly assumed $N/T\to 0$. The reality is more interesting when this does not hold. For instance, in finance and machine learning often $N\sim T$, then things get hairy. Take a look at one of the standard approaches to deal with this situation in this paper by Ledoit called shrinkage.
