5
$\begingroup$

I'm fitting a logistic regression model (with R's caret package) to data here. I aim to predict whether Hillary or Trump will win a given county.

The relevant code:

logisticSettings <- trainControl(method = "cv", number = 10, returnResamp = "all", classProbs = TRUE, summaryFunction = twoClassSummary)
logisticModel <- train(electTrain[,2:length(electTrain)], make.names(electTrain[,1]), method = "plr", metric = "ROC", trControl = logisticSettings)

electTrain is my training dataset; the first column is the column of classes and the rest is features. When I run this, I get the following error:

Error in solve.default(ddf) : 
system is computationally singular: reciprocal condition number = 9.55304e-17 

I think this stems at least in part from the data being highly correlated. For example, one column is 2010 population, and another is 2010 population estimate. To remedy this, I removed some columns from my training set so that no features were correlated at above .92 (arbitrary cutoff).

But the error persists. What's wrong? Some ideas:

  • The error cutoff is still too high.

  • One column is approximately a linear combination of two or more others.

  • I've made a mistake in the code.

$\endgroup$
6
$\begingroup$

Removing highly correlated (or identical) variables by hand can work, but :

  • It can become unfeasible as the number of variables becomes too large
  • Selecting the variables by hand is purely arbitrary
  • With factor variables, it becomes slightly harder to detect correlated variables (unless you look at the predictors with dummy variables)
  • Singularity can also arise because a variable is a linear combination of other variables, which needs some further preprocessing to detect

I would recommend a ridge regression / Tikhonov regularization :

  • It makes the matrix always invertible introducing a penalty
  • If some of the variables are identical, they will receive the same weight
  • It is easily usable (and fast) using the R package glmnet
  • The penalization parameter can be selected by cross-validation
| cite | improve this answer | |
$\endgroup$
2
$\begingroup$

RUser4512 has a good answer (+1). I just want to add some comments on Matrix Condition Number, which we can use to check the numerical stability issues. In R the function is kappa.

Here is an example in R. In this example, we create a data with two highly correlated columns. x1 and x2. Note they are not identical but really close.

In experiment 1, although it has a large condition number. R still can solve it.

> set.seed(0)
> x1=runif(1e3)*1e3
> x2=x1+runif(1e3)*1e-3
> x=cbind(x1,x2)
> y=runif(1e3)

> kappa(t(x) %*% x)
[1] 8.855766e+12

> solve(t(x) %*% x, t(x) %*%y)
        [,1]
x1 -399.9371
x2  399.9375

In experiment 2, we further reduce the difference on two columns by 1000 times. R will produce an error as you described.

> x[,2]=x[,2]*1e-3
> kappa(t(x) %*% x)
[1] 2.220277e+18
> solve(t(x) %*% x, t(x) %*%y)
Error in solve.default(t(x) %*% x, t(x) %*% y) : 
  system is computationally singular: reciprocal condition number = 4.49945e-19
| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.