Question regarding machine learning models in production

Question

for example, i have a feature with 5 distinct values and once one hot encoded this becomes 5 columns, but lets say the data that needs to be predicted has 4 distinct values, the neural network won't accept the data as it is not in the right dimensions.

How do I go about solving this issue? would I use a label encoder instead of one hot encoding?

Thanks for your help.

Update:

I realised my mistake i one hot encoded a feature that is not categorical in nature. parch Number of Parents/Children Aboard --- which makes sense that the test data can contain less or more values in the feature leading to issues when one hot encoding.

Answer 1

That's not really a problem if you approach this in a valid way.

By valid way I mean using training data to encode categories. If your test data has less categories than training data, then you'd just see less categories in test data. Encoding won't be a problem (contrast that with the situation where you have category in test unseen in training data).

To be precise I mean that you should use the same one-hot encoding for train and test data. Then both train and test one-hot encoded features would have 5 dimensions, so there is no problem.

Also, if you're not interested in the class that is in training, but not in test, you can just drop it from training data (but it might make model perform worse, so it should be compared to leaving it as is).

By the way, one alternative of label/one-hot encoding that also can be used for categories that are unseen in train data is target encoding (also called mean encoding). It consists of encoding categories as means of targets for examples that have this category. You can easily incorporate new category by encoding it with the mean value of target. I encourage you to see this video and its follow-ups for details and problems that target encoding faces.