Suppose I am training a linear model. What are the conceptual differences between using a diagonal covariance matrix and the identity? It is clear to me that the difference between a full covariance matrix and a diagonal covariance matrix is that there is no correlation between predictors with the diagonal matrix. I'm not quite sure of the differences between the identity and diagonal matrices though.
$\begingroup$
$\endgroup$
1
-
$\begingroup$ I would like to find the "decision surface" in feature space between those two classes. covariances equal to the identity matrix, $\endgroup$– Mohammed FayyadhMar 26, 2021 at 8:33
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
1
An identity covariance matrix, $\Sigma=I$ has variance = 1 for all variables.
A covariance matrix of the form, $\Sigma=\sigma^2I$ has variance = $\sigma^2$ for all variables.
A diagonal covariance matrix has variance $\sigma^2_i$ for the $i^\text{th}$ variable.
(All three have zero covariances between variates)
-
$\begingroup$ What effect does this have on the coefficients that I produce if I am using generalized linear regression to train, say, a multinomial linear model. What effect does each predictor having a different variance have? $\endgroup$– HXSP1947Sep 22, 2014 at 4:47
$\begingroup$
$\endgroup$
An identity matrix is by definition a matrix with 1's on the diagonal and 0's elsewhere. If you choose to use an identity matrix as your covariance matrix, then you are totally ignoring the data for calculating the variances. Is that really what you mean to do? The only way that could make sense is if you had already standardized the data to have variance 1.
