I would like to use categorical features to train an autoencoder neural network. Typically for categorical features my approach has been to one-hot encode the features to make indicator features. The trouble I have had though has been that for large data sets one-hot encoding often results in creating many features, some of which may be sparsely populated. I also am aware of techniques such as 'bucketing' the bottom 10% (for example) into an 'other' bucket, so that categories in a categorical feature that occur only once or twice all get bucketed into the same one-hot encoded feature.
After reading this paper I have become interested in the idea of using frequency distributions to represent categorical features. In other words each categorical feature is transformed such that each category (or value) in the feature is transformed to the probability of that category occurring in the feature. Hence the column now represents a probability distribution of the categories in the categorical feature.
My question: Does it make sense to transform categorical features in this way? My impression is that since I am using a neural network this approach is appropriate, but for distance or density based algorithms this approach would not be suitable as probability distributions shouldn't be evaluated spatially (as with a distance or density metric).
I have read the below posts, but they do not discuss using a probability distribution.