17
$\begingroup$

I am trying to make predictions using a random forest model in R.

However I get errors since some factors have different values in the test set than in the training set. For example, a factor Cat_2 has values 34, 68, 76, etc., in the test set that do not appear in the training set. Unfortunately, I do not have control over the Test set... I must use it as-is.

My only workaround was to convert the problematic factors back to numerical values, using as.numeric(). It works but I am not very satisfied, since these values are codes that have no numerical sense...

Do you think there would be another solution, to drop the new values from the test set? But without removing all the other factor values (let say values 1, 2, 14, 32, etc.) which are in both training and test, and contains information potentially useful for predictions.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ I see know reason why values in the test would have to be in the training set. The idea of classification is to use the training data to get an idea of what the class-conditional densities look like. You don't get to see every possible value from the density. I a variable is used in a split ona tree then the split determines which branch to follow for any unseen values as well as those that have been seen. $\endgroup$
    – Michael R. Chernick
    Commented May 30, 2012 at 0:04
  • $\begingroup$ You make a valid point, but on a practical level using the specific tool enquired about (the RF package in R) this is not allowed. My answer involving imputation is one way around it, though certainly not the best solution. Is does at least make the code not crash, so at least works, for small values of work. $\endgroup$
    – Bogdanovist
    Commented May 30, 2012 at 2:13
  • $\begingroup$ Similar to my question here: stats.stackexchange.com/questions/18004/…. I think I might use GBM instead of RF as it seems to deal with new factor levels better. Also, have you looked at the implementation of RF in party? I have never liked randomForest because of these issues (and inability to seamlessly deal with missing values). $\endgroup$
    – B_Miner
    Commented May 31, 2012 at 1:13

7 Answers 7

Reset to default
2
$\begingroup$

If the test set has a lot of these points with new factor values then I'm not sure what the best approach is. If it is just a handful of points you might be able to get away with something fudgy like treating the errant factor levels as missing data and imputing them with whatever approach you see fit. The R implementation has a couple of ways to impute missing data, you just need to set these factor levels to NA to indicate they are missing.

Improve this answer
$\endgroup$
8
$\begingroup$

King and Bonoit, this snippet can be useful to harmonize levels:

for(attr in colnames(training))
{
  if (is.factor(training[[attr]]))
  {
    new.levels <- setdiff(levels(training[[attr]]), levels(testing[[attr]]))
    if ( length(new.levels) == 0 )
    { print(paste(attr, '- no new levels')) }
    else
    {
      print(c(paste(attr, length(new.levels), 'of new levels, e.g.'), head(new.levels, 2)))
      levels(testing[[attr]]) <- union(levels(testing[[attr]]), levels(training[[attr]]))
    }
  }
}

It also prints which attributes are changed. I did not find a good way to write it more elegantly (with ldply or something). Any tips are appreciated.

Improve this answer
$\endgroup$
4
$\begingroup$

Here's some code I wrote that addresses @King's response above. It fixed the error:

# loops through factors and standardizes the levels
for (f in 1:length(names(trainingDataSet))) {
    if (levels(testDataSet[,f]) > levels(trainingDataSet[,f])) {    
            levels(testDataSet[,f]) = levels(trainingDataSet[,f])       
    } else {
            levels(trainingDataSetSMOTEpred[,f]) = levels(testDataSet[,f])      
    }
}
Improve this answer
$\endgroup$
1
  • $\begingroup$ hi @ifarb, I'm trying to understand your solution: what's trainingDataSetSMOTEpred and where's it defined in the code? $\endgroup$
    – Kasia Kulma
    Commented Jan 26, 2017 at 13:59
4
$\begingroup$

Test and training set should be combined as one set and then change the levels of the training set. My codes are:

totalData <- rbind(trainData, testData)
for (f in 1:length(names(totalData))) {
  levels(trainData[, f]) <- levels(totalData[, f])
}

This works in any cases where the number of levels in test are more or less than training.

Improve this answer
$\endgroup$
2
$\begingroup$

I have a lousy workaround when I use randomForest in R. It's probably not theoretically sound, but it gets the thing running.

levels(testSet$Cat_2) = levels(trainingSet$Cat_2)

or the other way round. Basically, it just gives tells R that it's a valid value just that there are 0 cases; so stop bugging me about the error.

I'm not smart enough to code it such that it automatically performs the action for all categorical features. Send me the code if you know how...

Improve this answer
$\endgroup$
1
  • $\begingroup$ But this does not work if the number of levels in test are more than training. It only works if test data factor levels are <= training data factor levels. $\endgroup$
    – KarthikS
    Commented Jan 15, 2016 at 2:22
2
$\begingroup$

If your test data has some levels that are an insignificant percentage of the population, but are just messing up your ability to make the model, here is an elegant way to remove the extraneous levels from the test set:

uniquetrain <- unique( train$category)
test <- test[test$category %in% uniquetrain,]
Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ For future readers, this doesn't simply remove the levels, it removes all of the rows that have a vale of category not in train$category. Whether or not this makes any sense for your problem depends on your goals, but I wouldn't recommend it in general. In particular, this solution isn't helpful if we care about the novel levels. $\endgroup$
    – Sycorax ♦
    Commented Nov 5, 2020 at 22:58
  • $\begingroup$ Correct. It would only be useful if those extra rows are irrelevant to the data you are analyzing, and are just getting in the way. (That was actually how this got developed) $\endgroup$
    – Lamden
    Commented Nov 5, 2020 at 23:02
1
$\begingroup$

I'm sure that you would have thought of this already if this were the case, but if the test set has actual values and you're using the test set for cross validation purposes, then re-splitting the dataframe into training and test datamframes where the two are balanced on these factors would avoid your problem. This method is popularly known as stratified cross-validation.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.