Handling missing/rare levels in predictor in data samples

Question

Let us assume we have a dataset with one catigorical variable, which is represented in R as a factor. I am performing crossvalidation to assess models, for which I need to perform stratified sampling of the data based on the target class.

In some of the folds for that factor, one or more levels do not occur, because those levels are rare, and therefore the prediction fails a lot of time when the out-of-bag data set is produced in the prediction phase, the models complain that new levels are generated.

Previously I have by force assigned empty levels in the resulting models and tried to make sure that the factors and the corresponding automatically assigned numeric values match-up.

Please keep in mind the perspective from which I want to perform the experiments which is as follows.

Condition: I am interested on comparative classifier performance evaluation using benchmark datasets, and not to create a model to use it in actual prediction of data.

Question1: How this can be handled.

I am aware that we should possibly bin these kind of rare labels together, or group very rare labels as "others" based on the domain knowledge of the data or from the analysis. Shall this be done to benchmark datasets? The datasets in question are multi-label datasets used in the context of Label Powerset method . Although, I have observed this identical problem when cross-validating some other multi-class real life datasets to, but those could be binned and manipulated.

Question2: When doing a classifier evaluation, shall I modify these standard benchmark datasets and somehow get rid of the rare labels, so that they do not interfere as mentioned above? Although, if using the same modified dataset when comparing a series of classifier types should be okay and comparable, I would like to know what other opinions others have on this.

In literature, either

• just trained using a train set and produced a result on the test set
• used test set as validation set, and no test set. Then reported both train/test scores.
• manually created folds, and used that single fold division to train a classifier multiple times and take the mean stats.

Question3: From the perspective of comparative classifier performance, how can I use the separated train and test sets (which does not have the problems with factor levels) to estimate regularized performance? For example, although does not make sense from the point of view of testing on new data, but, will using the test data as the validation dataset do any good?

Answer 1

Answering your questions directly:

Question 1. How this can be handled.

I've also encountered the situation you described where I created multiple random splits of training, validation, and test. Due to random sampling, there may be some discrete variable value (for either numeric or categorical variables) that does not occur in training, so the trained model does not incorporate it. However, the value may occur in either the validation or test splits, and when predict() is called, the model will choke when it sees the value not previously seen during training. The end result is that predict() will fail with an error message.

A quick way to determine if you have a rare value is to run:

table(dataframe$col)

     1      2      3      4      5      6 
157559   4645  17679   1448     10 117585 

Above, you can see that the value 5 is rare for this variable.

How I've handled this situation is to simply remove all rows with that value from the dataset. Since these rows are removed from the original dataset, they are removed from all the splits that will be created.

dataframe <- subset(dataframe, col != 5)

This should be fine in your case because, as you wrote, you are trying to compare algorithms rather than use the model to make predictions in a production environment. In the latter case, one can imagine in the future that there may be a rare situation where you could get a new data instance that actually contains the rare value, then you are in trouble.

Question 2. When doing a classifier evaluation, shall I modify these standard benchmark datasets and somehow get rid of the rare labels, so that they do not interfere as mentioned above?

Yes, I wrote above. Other folks may have different opinions. If you are publishing results, you will want to state that you pre-processed the data set to remove these outlier values.

Question 3: From the perspective of comparative classifier performance, how can I use the separated train and test sets (which does not have the problems with factor levels) to estimate regularized performance? For example, although does not make sense from the point of view of testing on new data, but, will using the test data as the validation dataset do any good?

You should follow good procedure and create separate training, validation, and test data sets.

Here is what I would do if I were in your place:

1. Remove outlier values and do any other pre-processing, like scaling variables, on your original data set.

2. If you have sufficient data (something like 100,000 data instances), then create random training, validation, and test data sets using 60/20/20 splits.

3. Train models using the training data. For a given algorithm, tune the hyper-parameters by getting the best results when applying your model to the validation data set. Best results depend on your situation, whether it's accuracy, RMSE, F1, or whatever. Repeat this step for each of the N algorithms to get the best hyperparameters for each one. Now you have N trained models.

4. Apply the N trained models on the test data set. You can select the model that gives you best results on this set.

For more information on the proper procedure of how to use separate training, validation, and test data sets, see Andrew Ng's video lecture series. On Coursera, you should watch the entire "Advice for Applying Machine Learning (Week 6)" section. The videos are also found on Youtube.