let's assume an input dataset that is a mix of categorical values and real values. When preprocessing this data into an appropriate NN input, OHE is recommended because it doesn't assume any order of the categories. ["Man", "Woman", "Diverse"] has no order to it so having one input that represents them all within one dimension makes little sense.
When using cross validation, the dataset often gets split into a lot smaller subsets. These subsets may not hold all categories. When using OHE of sklearn, the input set is used to determine the dimensionality. This can lead to unpredictable column counts of the networks data input. It can also lead to different categories taking different positions in the NN.
How would one process this input to feed to the NN without hard-coding all possible categories and still be able to handle varying numbers of categories in the input set?
Two intuitive ideas that don't work:
- determine the dataset size (after OHE) and set the input size of the NN based off of that: Doesn't work because each CV subset would potentially have a different model and each category position in the dimensions doesn't map to other subsets
- 0 pad to an arbitrarily high (but likely never reached) input size: dirty, because the ordering is still not ensured and if one category is missing, all inputs may be shifted by 1+ positions
Would autoencoders help?
Edit: This is a cross post because I realized that there is a really hard to understand overlap between the subdomains AI.stack.., datascience.stackex... and stats.stoackex..., all focusing on machine learning, AI and datascience. This seems to be the most popular subpage and I went ahead and posted it here as well. Surprisingly, I did get answers on the datascience post very quickly.