# Collinear variables in Multiclass LDA training

I'm training a Multi-class LDA classifier with 8 classes of data.

While performing training, I get a warning of: "Variables are collinear"

I'm getting a training accuracy of over 90%.

I'm using scikits-learn library in Python do train and test the Multi-class data.

I get decent testing accuracy too(about 85%-95%).

-

Because LDA, like regression techniques involves computing a matrix inversion, which is inaccurate if the determinant is close to 0 (i.e. two or more variables are almost a linear combination of each other).

More importantly, it makes the estimated coefficients impossible to interpret. If an increase in $X_1$, say, is associated with an decrease in $X_2$ and they both increase variable $Y$, every change in $X_1$ will be compensated by a change in $X_2$ and you will underestimate the effect of $X_1$ on $Y$. In LDA, you would underestimate the effect of $X_1$ on the classification.

If all you care for is the classification per se, and that after training your model on half of the data and testing it on the other half you get 85-95% accuracy I'd say it is fine.

-
So can I interpret this as, a feature X1 in the feature vector is not a good pick in case the testing accuracy is low? –  raul_w May 29 '12 at 23:33
I guess that if testing accuracy is low there no good pick. –  gui11aume May 30 '12 at 8:08
Whats interesting is I'm having this problem with LDA but not when I use QDA. I wonder whats different in there? –  raul_w May 30 '12 at 22:55