7
$\begingroup$

thanks in advance for the help.

Suppose I am training a linear model. What are the conceptual differences between using a diagonal covariance matrix and the identity? It is clear to me that the difference between a full covariance matrix and a diagonal covariance matrix is that there is no correlation between predictors with the diagonal matrix. I'm not quite sure of the differences between the identity and diagonal matrices though.

Improve this question
$\endgroup$
10
$\begingroup$

An identity covariance matrix, $\Sigma=I$ has variance = 1 for all variables.

A covariance matrix of the form, $\Sigma=\sigma^2I$ has variance = $\sigma^2$ for all variables.

A diagonal covariance matrix has variance $\sigma^2_i$ for the $i^\text{th}$ variable.

(All three have zero covariances between variates)

Improve this answer
$\endgroup$
  • $\begingroup$ What effect does this have on the coefficients that I produce if I am using generalized linear regression to train, say, a multinomial linear model. What effect does each predictor having a different variance have? $\endgroup$ – HXSP1947 Sep 22 '14 at 4:47
  • $\begingroup$ What do predictors have to do with your question? You condition on predictors in regression models. $\endgroup$ – Glen_b Sep 22 '14 at 5:59
1
$\begingroup$

An identity matrix is by definition a matrix with 1's on the diagonal and 0's elsewhere. If you choose to use an identity matrix as your covariance matrix, then you are totally ignoring the data for calculating the variances. Is that really what you mean to do? The only way that could make sense is if you had already standardized the data to have variance 1.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.