0
$\begingroup$

for example, i have a feature with 5 distinct values and once one hot encoded this becomes 5 columns, but lets say the data that needs to be predicted has 4 distinct values, the neural network won't accept the data as it is not in the right dimensions.

How do I go about solving this issue? would I use a label encoder instead of one hot encoding?

Thanks for your help.

Update:

I realised my mistake i one hot encoded a feature that is not categorical in nature. parch Number of Parents/Children Aboard --- which makes sense that the test data can contain less or more values in the feature leading to issues when one hot encoding.

Improve this question
$\endgroup$
1
  • $\begingroup$ Your encoding is correct. Your input has 5 columns in this case and your output should have 4. How you choose to go from these 5 columns to 4 is the choice of your neural network/approach. $\endgroup$
    – Jan
    Commented Oct 17, 2018 at 8:44

1 Answer 1

Reset to default
0
$\begingroup$

That's not really a problem if you approach this in a valid way.

By valid way I mean using training data to encode categories. If your test data has less categories than training data, then you'd just see less categories in test data. Encoding won't be a problem (contrast that with the situation where you have category in test unseen in training data).

To be precise I mean that you should use the same one-hot encoding for train and test data. Then both train and test one-hot encoded features would have 5 dimensions, so there is no problem.

Also, if you're not interested in the class that is in training, but not in test, you can just drop it from training data (but it might make model perform worse, so it should be compared to leaving it as is).

By the way, one alternative of label/one-hot encoding that also can be used for categories that are unseen in train data is target encoding (also called mean encoding). It consists of encoding categories as means of targets for examples that have this category. You can easily incorporate new category by encoding it with the mean value of target. I encourage you to see this video and its follow-ups for details and problems that target encoding faces.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Hi Jakub that was my intuition as well but i came across this issue with the titanic data set. I used the same one hot encoding for both the train and test data, however the test data contained a value that was not there in the training data. Hence the test data had one extra feature meaning the neural network could not accept the data. To solve this problem i did not one hot encode that specific feature. $\endgroup$
    – A.H.
    Commented Oct 17, 2018 at 10:57
  • $\begingroup$ That's not the problem you described in the question, it's exactly the opposite $\endgroup$
    – Jakub Bartczuk
    Commented Oct 18, 2018 at 8:15
  • $\begingroup$ Yes it is the complete opposite but if it had less or more values than the training data wouldn't make a difference, dimensions would still be wrong after one hot encoding :) $\endgroup$
    – A.H.
    Commented Oct 18, 2018 at 10:29
  • $\begingroup$ No, they would not if it was done correctly. You can check it yourself using any package that lets you train onehot encoder, for example scikit-learn $\endgroup$
    – Jakub Bartczuk
    Commented Oct 18, 2018 at 10:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.