Suppose I am training a linear model. What are the conceptual differences between using a diagonal covariance matrix and the identity? It is clear to me that the difference between a full covariance matrix and a diagonal covariance matrix is that there is no correlation between predictors with the diagonal matrix. I'm not quite sure of the differences between the identity and diagonal matrices though.

  • $\begingroup$ I would like to find the "decision surface" in feature space between those two classes. covariances equal to the identity matrix, $\endgroup$ Mar 26 at 8:33

An identity covariance matrix, $\Sigma=I$ has variance = 1 for all variables.

A covariance matrix of the form, $\Sigma=\sigma^2I$ has variance = $\sigma^2$ for all variables.

A diagonal covariance matrix has variance $\sigma^2_i$ for the $i^\text{th}$ variable.

(All three have zero covariances between variates)

  • $\begingroup$ What effect does this have on the coefficients that I produce if I am using generalized linear regression to train, say, a multinomial linear model. What effect does each predictor having a different variance have? $\endgroup$
    – HXSP1947
    Sep 22 '14 at 4:47
  • $\begingroup$ What do predictors have to do with your question? You condition on predictors in regression models. $\endgroup$
    – Glen_b
    Sep 22 '14 at 5:59

An identity matrix is by definition a matrix with 1's on the diagonal and 0's elsewhere. If you choose to use an identity matrix as your covariance matrix, then you are totally ignoring the data for calculating the variances. Is that really what you mean to do? The only way that could make sense is if you had already standardized the data to have variance 1.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.