# Collinearity and Linear Discriminant Analysis

I trying to conduct linear discriminant analysis using the lda package and I keep getting a warning message saying that the variables are collinear.

I want to pinpoint and remove the redundant variables. What is the best method for doing this in R?

I've read about solutions such as stepwise selection which can be used to do this but this doesn't work with discriminant analysis.

I tried lasso regression but this shrank my 66 variables down to just 12 - the optimal set and it's hard to identify the order in which it's done this as I would prefer to keep a larger number.

If the variables are numeric you can use correlation and then clustering to pair all the variables highly correlated to each other. The code below will cluster variables which have correlation coefficient greater than 0.95

Lets say df is your matrix

corDF = cor(df);
dissimilarity <- 1 - abs(corDF);
distance <- as.dist(dissimilarity);
hc <- hclust(distance);
clusterV = cutree(hc,h=0.05);


clusterV is an array with as many elements as variables in your data. The values represent the cluster the variable belongs to.

• Hi @DeepakML. Can you please describe further what the values in the clusterV array refer to? If 2 variables have the value '1', for example, does this mean they are collinear? – J.Con Aug 22 '17 at 5:44

If the collinearity is rather simple, i.e. pairs of variables are tightly correlated then you can run cor over all the pairwise combinations. A more efficient way is to look at rcorr in the Hmisc package.

> rcorr(matrix( c(1:10,1:10,runif(10)), ncol=3) )
[,1]  [,2]  [,3]
[1,]  1.00  1.00 -0.16
[2,]  1.00  1.00 -0.16
[3,] -0.16 -0.16  1.00

n= 10

P
[,1]   [,2]   [,3]
[1,]        0.0000 0.6624
[2,] 0.0000        0.6624
[3,] 0.6624 0.6624


The first matrix above is the correlation matrix. So columns 1 and 2 are perfectly correlated.

If the collinearity is more complex (as often happens with factor variables where three or more form dependent linear combinations) than you may need to use true matrix methods. You can get the rank of a matrix using the qr function. It appears to me with a small test that the pivoting is done to move some of "dependent" or non-independent variables to the right side (noticing the $qraux value of 0 for column 2): qr(matrix( c(1:3,1:3,runif(3)), 3) )$qr
[,1]       [,2]      [,3]
[1,] -3.7416574 -1.1696086 -3.741657
[2,]  0.5345225  0.6603587  0.000000
[3,]  0.8017837  0.9623882  0.000000

$rank [1] 2$qraux
[1] 1.267261 1.271678 0.000000

\$pivot
[1] 1 3 2

attr(,"class")
[1] "qr"


Other than eyeballing structures from correlation matrix, there are 3 better ways to do it, especially when your dimensionality is higher than what you could hand-pick features from.

1. variance inflation factor (VIF).

https://www.statsmodels.org/dev/generated/statsmodels.stats.outliers_influence.variance_inflation_factor.html

instead of showing the correlation of a feature to every other feature, VIF gives you the collinearity between one feature and a linear model made of all the other features. you can easily set up a threshold for the VIF value and exclude the features that doesn't give more information to the linear combination of other features.

1. regularization / shrinkage

https://scikit-learn.org/stable/auto_examples/classification/plot_lda.html#sphx-glr-auto-examples-classification-plot-lda-py

The bad thing about collinearity is that it makes the within-class covariance matrix close to singular matrix, resulting in impossibility or inaccuracy of calculating inverse matrix. This problem can be circumvented by having a shrinkage, i.e. averaging the covariance matrix with a diagonal matrix. It also helps the problem of having small sample size and prevents overfitting.

1. PCA before LDA

https://www.sciencedirect.com/science/article/pii/S0031320302000481

it has been a common practice. After PCA the collinear features will be combined into components without losing information, if you pick enough components.