I am working on a Kaggle Housing Dataset (https://www.kaggle.com/c/house-prices-advanced-regression-techniques) to predict a house price using several regression techniques. For this project I am using R/Rstudio. However, I ran into a issue when dealing with categorical data. The problem is that most of the categorical features are extremely imbalanced. Here are some of the examples.


Artery  Feedr   Norm   PosA   PosN   RRAe   RRAn   RRNn 
     5     13   2889      4      4      1      1      2 


1.5Fin 1.5Unf 1Story 2.5Fin 2.5Unf 2Story SFoyer   SLvl 
   314     19   1471      8     24    872     83    128 

The issue that this creates is that when I train-test-split, one of the data can include classes of a categorical feature that is not included in the other dataset. For example, train data can include "RPAn" and "RPAe" from the Condition2 feature, but test set do not include them.

This result in another issue where I train a model and test, it will return an error where the model could not find the classes that it was trained on for the test set. The error looks like this.

lm_model <- lm(SalePrice ~ ., pre_train)
lm_result <- predict(lm_model, newdata=validation) 

Error in model.frame.default(Terms, newdata, 
     na.action = na.action, xlev = object$xlevels) : 
  factor RoofMatl has new levels Membran, Roll

What would be the right approach to tackle this problem? Are the only way to just delete the categorical features that are extremely imbalanced(which ends up being majority of the categorical feature in the dataset)? If someone can answer this, that would be super helpful!


1 Answer 1


If your intent is prediction and not inference, there are a few ways around this I suppose.

One approach is to NOT make predictions for observations which have new levels. This is the approach common modelling libraries (like tidymodels) will take. It isn't a great approach in my opinion, but it is certainly one approach.

The other is to use domain knowledge to collapse some of the less prevalent groups into larger categories. Maybe it would make sense to group 2 Bed + Den units together with the 2 Bed units. Maybe it wouldn't. It's up to you.

Another solution would be to use a random effect rather than a fixed effect. When new categories appear, random effects can naturally handle them, as I demonstrate below

#> Registered S3 method overwritten by 'tune':
#>   method                   from   
#>   required_pkgs.model_spec parsnip
#> Loading required package: Matrix
#> Attaching package: 'Matrix'
#> The following objects are masked from 'package:tidyr':
#>     expand, pack, unpack

# Generate imbalanced data
N = 100
groups = sample(letters[1:3], size=N, replace = T, prob = c(0.95, 0.04, 0.01))
y = model.matrix(~groups-1) %*% c(2, 1, -1) + rnorm(N, 0, 0.24)
d = tibble(groups, y)

# Split up the data
splits = initial_split(d)
train_set = training(splits)
val_set = testing(splits)

#missing b!
train_set %>% 
#> # A tibble: 2 × 2
#>   groups     n
#>   <chr>  <int>
#> 1 a         73
#> 3 c          1

# has b!
val_set %>% 
#> # A tibble: 2 × 2
#>   groups     n
#>   <chr>  <int>
#> 1 a         25
#> 2 b          1

model = lmer(y~1 + (1|groups), data = train_set)

preds = predict(model, newdata=val_set, allow.new.levels = T)

Created on 2021-10-12 by the reprex package (v2.0.1)

Finally, if there are just one or two categories which don't appear frequently, you could just ignore them. This approach becomes problematic however when you have many categories appearing a few times and a handful making up the majority of the observations.

All in all, my recommendation would be to use domain knowledge to lump categories together if possible. If once you've done that and you can't lump categories together in a reasonable way, my recommendation would be to use a mixed effect model.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.