Let us assume we have a dataset with one catigorical variable, which is represented in R as a factor. I am performing crossvalidation to assess models, for which I need to perform stratified sampling of the data based on the target class.
In some of the folds for that factor, one or more levels do not occur, because those levels are rare, and therefore the prediction fails a lot of time when the out-of-bag data set is produced in the prediction phase, the models complain that new levels are generated.
Previously I have by force assigned empty levels in the resulting models and tried to make sure that the factors and the corresponding automatically assigned numeric values match-up.
Please keep in mind the perspective from which I want to perform the experiments which is as follows.
Condition: I am interested on comparative classifier performance evaluation using benchmark datasets, and not to create a model to use it in actual prediction of data.
Question1: How this can be handled.
I am aware that we should possibly bin these kind of rare labels together, or group very rare labels as "others" based on the domain knowledge of the data or from the analysis. Shall this be done to benchmark datasets? The datasets in question are multi-label datasets used in the context of Label Powerset method . Although, I have observed this identical problem when cross-validating some other multi-class real life datasets to, but those could be binned and manipulated.
Question2: When doing a classifier evaluation, shall I modify these standard benchmark datasets and somehow get rid of the rare labels, so that they do not interfere as mentioned above? Although, if using the same modified dataset when comparing a series of classifier types should be okay and comparable, I would like to know what other opinions others have on this.
In literature, either
- just trained using a train set and produced a result on the test set
- used test set as validation set, and no test set. Then reported both train/test scores.
- manually created folds, and used that single fold division to train a classifier multiple times and take the mean stats.
Question3: From the perspective of comparative classifier performance, how can I use the separated train and test sets (which does not have the problems with factor levels) to estimate regularized performance? For example, although does not make sense from the point of view of testing on new data, but, will using the test data as the validation dataset do any good?