# Linear Regression: Extremely Imbalanced Categorical Features

I am working on a Kaggle Housing Dataset (https://www.kaggle.com/c/house-prices-advanced-regression-techniques) to predict a house price using several regression techniques. For this project I am using R/Rstudio. However, I ran into a issue when dealing with categorical data. The problem is that most of the categorical features are extremely imbalanced. Here are some of the examples.

table(data$Condition2) Artery Feedr Norm PosA PosN RRAe RRAn RRNn 5 13 2889 4 4 1 1 2 table(data$HouseStyle)

1.5Fin 1.5Unf 1Story 2.5Fin 2.5Unf 2Story SFoyer   SLvl
314     19   1471      8     24    872     83    128


The issue that this creates is that when I train-test-split, one of the data can include classes of a categorical feature that is not included in the other dataset. For example, train data can include "RPAn" and "RPAe" from the Condition2 feature, but test set do not include them.

This result in another issue where I train a model and test, it will return an error where the model could not find the classes that it was trained on for the test set. The error looks like this.

lm_model <- lm(SalePrice ~ ., pre_train)
summary(lm_model)
lm_result <- predict(lm_model, newdata=validation)

Error in model.frame.default(Terms, newdata,
na.action = na.action, xlev = object\$xlevels) :
factor RoofMatl has new levels Membran, Roll


What would be the right approach to tackle this problem? Are the only way to just delete the categorical features that are extremely imbalanced(which ends up being majority of the categorical feature in the dataset)? If someone can answer this, that would be super helpful!

If your intent is prediction and not inference, there are a few ways around this I suppose.

One approach is to NOT make predictions for observations which have new levels. This is the approach common modelling libraries (like tidymodels) will take. It isn't a great approach in my opinion, but it is certainly one approach.

The other is to use domain knowledge to collapse some of the less prevalent groups into larger categories. Maybe it would make sense to group 2 Bed + Den units together with the 2 Bed units. Maybe it wouldn't. It's up to you.

Another solution would be to use a random effect rather than a fixed effect. When new categories appear, random effects can naturally handle them, as I demonstrate below

library(tidymodels)
#> Registered S3 method overwritten by 'tune':
#>   method                   from
#>   required_pkgs.model_spec parsnip
library(tidyverse)
library(lme4)
#>
#> Attaching package: 'Matrix'
#> The following objects are masked from 'package:tidyr':
#>
#>     expand, pack, unpack

# Generate imbalanced data
set.seed(0)
N = 100
groups = sample(letters[1:3], size=N, replace = T, prob = c(0.95, 0.04, 0.01))
y = model.matrix(~groups-1) %*% c(2, 1, -1) + rnorm(N, 0, 0.24)
d = tibble(groups, y)

# Split up the data
splits = initial_split(d)
train_set = training(splits)
val_set = testing(splits)

#missing b!
train_set %>%
count(groups)
#> # A tibble: 2 × 2
#>   groups     n
#>   <chr>  <int>
#> 1 a         73
#> 3 c          1

# has b!
val_set %>%
count(groups)
#> # A tibble: 2 × 2
#>   groups     n
#>   <chr>  <int>
#> 1 a         25
#> 2 b          1

model = lmer(y~1 + (1|groups), data = train_set)

preds = predict(model, newdata=val_set, allow.new.levels = T)


Created on 2021-10-12 by the reprex package (v2.0.1)

Finally, if there are just one or two categories which don't appear frequently, you could just ignore them. This approach becomes problematic however when you have many categories appearing a few times and a handful making up the majority of the observations.

All in all, my recommendation would be to use domain knowledge to lump categories together if possible. If once you've done that and you can't lump categories together in a reasonable way, my recommendation would be to use a mixed effect model.