I am working on a Kaggle Housing Dataset (https://www.kaggle.com/c/house-prices-advanced-regression-techniques) to predict a house price using several regression techniques. For this project I am using R/Rstudio. However, I ran into a issue when dealing with categorical data. The problem is that most of the categorical features are extremely imbalanced. Here are some of the examples.
table(data$Condition2) Artery Feedr Norm PosA PosN RRAe RRAn RRNn 5 13 2889 4 4 1 1 2 table(data$HouseStyle) 1.5Fin 1.5Unf 1Story 2.5Fin 2.5Unf 2Story SFoyer SLvl 314 19 1471 8 24 872 83 128
The issue that this creates is that when I train-test-split, one of the data can include classes of a categorical feature that is not included in the other dataset. For example, train data can include "RPAn" and "RPAe" from the Condition2 feature, but test set do not include them.
This result in another issue where I train a model and test, it will return an error where the model could not find the classes that it was trained on for the test set. The error looks like this.
lm_model <- lm(SalePrice ~ ., pre_train) summary(lm_model) lm_result <- predict(lm_model, newdata=validation) Error in model.frame.default(Terms, newdata, na.action = na.action, xlev = object$xlevels) : factor RoofMatl has new levels Membran, Roll
What would be the right approach to tackle this problem? Are the only way to just delete the categorical features that are extremely imbalanced(which ends up being majority of the categorical feature in the dataset)? If someone can answer this, that would be super helpful!