0
$\begingroup$

I have a data set with size of 30k+, and it has several categorical features and also several numerical features. When I try to split the data into training/testing data set, I need to answer a question, which is:

  1. For categorical features, how to make sure that training data has all the different categorical values.
  2. If not( there are categorical values only exist in test data), does it mean the trained model is bad model, because it won`t understand those values not from training data.

Context: I am training the decision tree model.

$\endgroup$
0
$\begingroup$

You could stratify your data such that the training data has all the categorical values. This can become rather tedious if you have many variables. If you only do this for one binary variable, then you simply create 2 subsets (1 set for case FALSE, 1 set for case TRUE), randomly split each into training and test sets, and then merge the training and test parts. I don't know easy methods for stratifying for many variables but this one will achieve what you want. There, they create groups by minimizing differences in mean and variance of all variables in training and test set. This will require quite some work to implement (the authors provide some code).

If you completely randomize (i.e., not stratifying for variables) the distribution of variables should be the same between training and test set. This only works when you many samples - you have 30k and 'several' variables, so that shouldn't be a problem. If it is still a problem, this means that some of those categories are relatively rare. Then you should ask yourself whether those categories are at all useful for predictions: if they only seldom occur, is it worthwhile even considering them? It could of course be that these categories are very predictive of an outcome, then then they still useful and you want to use methods as I mentioned above.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.