I have 12,000 records and I"d like to predict a two-class outcome. I'm deciding which predictors to keep and I'm having trouble with two problems.

1- I get an error message because I have categories with too few values.

2- I get an error message because of collinearity.

I'll go through both and speak more to the dataset. First, the dataset has 8,000 rows for training data, with 3 numeric predictors and the rest categoric.

The first issue occurs since some of the categoric variables have many groups and some of them end up with few values in the training set. I get this error allong with a long list of variables: task 1 failed - "variable appears to be constant within groups".

I used this code to solve for that --

#create dummy variables in my data frame
dummyVars <- dummyVars(~., data = df[,c(predictors, outcomeName)])
dummyDF <- as.data.frame(predict(dummyVars, df[,c(predictors, outcomeName)]))

#split training and testing data
index <- createDataPartition(df$outcome, p=.70, list=FALSE)
trainSet <- df[ index,]
testSet <- df[-index,]

#eliminate categories with fewer than 30 records
x <- NULL
z <- NULL
i <- 1
while (i < length(trainSet)){
  x <- c(x, i)
  z <- c(z, sum(trainSet[i]))
  i <- i+1

y <- data.frame(colnum = x, value = z) 
namesToExclude <- names[y[y$value>=0 &y$value<30,]$colnum]    
predictorsProcessed <- names[!(names%in%namesToExclude | names==outcomeName) | names%in%numerics]

#use rfe to get a list of the best predictors
ctrl <- rfeControl(functions = ldaFuncs, method = "cv")
results2 <- rfe(trainSet[,predictorsProcessed], as.factor(trainSet[,outcomeName]), sizes=c(1:100), rfeControl=ctrl)

That leads to the second problem, which is this error message: In lda.default(x, grouping, ...) : variables are collinear.

I know how to check for collinearity on the variable-wide level. I did something like that when I ran this code --

glm.fit1 <- glm(as.factor(outcome)~ 
                  category1 + category2 + category3 + category4, family = binomial, data = trainSet[,c(predictors,outcomeName)])


Here's my problem: these VIF scores give me one score for each variable before they are encoded. EG, it says user_type VIF: 2.85 whereas my variables that are used in rfe() are encoded as user_typeA, user_typeB and user_typeC with one-hot encoding.

How do I find collinearity for my variables after they've been encoded?

  • $\begingroup$ This means you have close to perfect colinearity? a VIF value of 2.85 should be ok i'd say. $\endgroup$ – Tom Mar 11 at 23:19
  • $\begingroup$ Those are the VIF scores before the predictors get split into categories with one-hot encoding. I think it is the categories that are perfectly colinear. $\endgroup$ – Cauder Mar 11 at 23:36
  • $\begingroup$ OK got it, that's because you should drop a column then in your categories. For instance, if you have a binary class (with categories 1 and 0), then if you use two columns for the 0 and the 1. Obviously you can exactly predict the one from the other. $\endgroup$ – Tom Mar 12 at 10:47
  • $\begingroup$ Thanks, that did it. What happens to the category that doesn't get included? EG, say I have a categoric with three classes. I include A and B, exclude C. In a model with one excluded class, I assume it's represented by the y-intercept. What happens when you have multiple categoric variables that are each excluding one class? $\endgroup$ – Cauder Mar 13 at 0:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.