# Collinear variables in Multiclass LDA training

I'm training a Multi-class LDA classifier with 8 classes of data.

While performing training, I get a warning of: "Variables are collinear"

I'm getting a training accuracy of over 90%.

I'm using scikits-learn library in Python do train and test the Multi-class data.

I get decent testing accuracy too(about 85%-95%).

Because LDA, like regression techniques involves computing a matrix inversion, which is inaccurate if the determinant is close to 0 (i.e. two or more variables are almost a linear combination of each other).

More importantly, it makes the estimated coefficients impossible to interpret. If an increase in $X_1$, say, is associated with an decrease in $X_2$ and they both increase variable $Y$, every change in $X_1$ will be compensated by a change in $X_2$ and you will underestimate the effect of $X_1$ on $Y$. In LDA, you would underestimate the effect of $X_1$ on the classification.

If all you care for is the classification per se, and that after training your model on half of the data and testing it on the other half you get 85-95% accuracy I'd say it is fine.

• So can I interpret this as, a feature X1 in the feature vector is not a good pick in case the testing accuracy is low? – garak May 29 '12 at 23:33
• I guess that if testing accuracy is low there no good pick. – gui11aume May 30 '12 at 8:08
• Whats interesting is I'm having this problem with LDA but not when I use QDA. I wonder whats different in there? – garak May 30 '12 at 22:55
• +1 for the answer, but " computing a matrix inversion" may not be accurate. We never computer it explicitly, direct methods such as LU, QR or iterative methods, are used. – Haitao Du Mar 20 '18 at 17:56
• @hxd1011 Correct! For the record, could you either give a few words about what happens in LU / QR etc. when the matrix is "almost singular", or perhaps point to a document that explains it? – gui11aume Mar 21 '18 at 12:41

As I seem to think gui11aume has given you a great answer, I want to give an example from a slightly different angle that might be illuminating. Consider that a covariate in your discriminant function looks as follows:

$X_1= 5X_2 +3X_3 -X_4$.

Suppose the best LDA has the following linear boundary:

$X_1+2X_2+X_3-2X_4 =5$

Then we can substitute $5X_2+3X_3-X_4$ for $X_1$ n the LDA boundary equation, so:

$5X_2+3X_3-X_4+2X_2+X_3-2X_4=5$

or

$7X_2+4X_3-3X_4=5$.

These two boundaries are identical but the first one has coefficients $1, 2, 1,-2$ for $X_1$, $X_2$, $X_3$, and $X_4$ respectively, while the other has coefficients $0, 7, 3, -1$.

So the coefficient are quite different but the two equations give the same boundary and identical prediction rule. If one form is good the other is also. But now you can see why gui11ame says the coefficients are uninterpretable.

There are several other ways to express this boundary as well by substituting for $X_2$ to give it the $0$ coefficient and the same could be done for $X_3$ or $X_4$. But in practice the collinearity is approximate. This makes things worse because the noise allows for a unique answer. Very slight perturbations of the data will cause the coefficients to change drastically. But for prediction you are okay because each equation defines almost the same boundary and so LDA will result in nearly identical predictions.

While the answer that was marked here is correct, I think you were looking for a different explanation to find out what happened in your code. I had the exact same issue running through a model.

Here's whats going on: You're training your model with the predicted variable as part of your data set. Here's an example of what was occurring to me without even noticing it:

df = pd.read_csv('file.csv')
df.columns = ['COL1','COL2','COL3','COL4']
train_Y = train['COL3']
train_X = train[train.columns[:-1]]


In this code, I want to predict the value of 'COL3'... but, if you look at train_X, I'm telling it to retrieve every column except the last one, so its inputting COL1 COL2 and COL3, not COL4, and trying to predict COL3 which is part of train_X.

I corrected this by just moving the columns, manually moved COL3 in Excel to be the last column in my data set (now taking place of COL4), and then:

df = pd.read_csv('file.csv')
df.columns = ['COL1','COL2','COL3','COL4']
train_Y = train['COL4']
train_X = train[train.columns[:-1]]


If you don't want to move it in Excel, and want to just do it by code then:

df = pd.read_csv('file.csv')
df.columns = ['COL1','COL2','COL3','COL4']
train_Y = train['COL3']
train_X = train[train.columns['COL1','COL2','COL4']]


Note now how I declared train_X, to include all columns except COL3, which is part of train_Y.

I hope that helps.