I am getting the "New factors levels not present in training data" error. But I checked the nlevels and class for every column in development as well as test data and they are the same. Any plausible explanation?
RF handles factors by one-hot encoding them. It makes one new dummy column for every level of the factor variable. When there are new or different factor levels in a scoring dataframe, bad things happen.
If the train and test existed together in the same data structure at the point that the factor was defined, there isn't a problem. When the test has its factor defined separately then you get issues.
library("randomForest") # Fit an RF on a few numerics and a factor. Give test set a new level. N <- 100 df <- data.frame(num1 = rnorm(N), num2 = rnorm(N), fac = sample(letters[1:4], N, TRUE), y = rnorm(N), stringsAsFactors = FALSE) df[100, "fac"] <- "a suffusion of yellow" df$fac <- as.factor(df$fac) train <- df[1:50, ] test <- df[51:100, ] rf <- randomForest(y ~ ., data=train) # This is fine, even though the "yellow" level doesn't exist in train, RF # is aware that it is a valid factor level predict(rf, test) # This is not fine. The factor level is introduced and RF can't know of it test$fac <- as.character(test$fac) test[50, "fac"] <- "toyota corolla" test$fac <- as.factor(test$fac) predict(rf, test)
You can get around this issue by relevelling your scoring factors to match the training data.
# Can get around by relevelling the new factor. "toyota corolla" becomes NA test$fac <- factor(test$fac, levels = levels(train$fac)) predict(rf, test)
I just encountered the issue as well when using
expand.grid() to examine predictions of
randomForest() across various factor levels.
The issue is created by
stringsAsFactors = T by default, which coerces strings to factors using the available levels of the data. This creates a problem when one is only using a subset of factor levels for predictions.
I fixed the issue by setting
stringsAsFactors = F which then allows
randomForest() to do the one hot encoding as the previous answer suggested.