Can I use a frequency distribution to encode categorical variables in a neural network?

Question

I would like to use categorical features to train an autoencoder neural network. Typically for categorical features my approach has been to one-hot encode the features to make indicator features. The trouble I have had though has been that for large data sets one-hot encoding often results in creating many features, some of which may be sparsely populated. I also am aware of techniques such as 'bucketing' the bottom 10% (for example) into an 'other' bucket, so that categories in a categorical feature that occur only once or twice all get bucketed into the same one-hot encoded feature.

After reading this paper I have become interested in the idea of using frequency distributions to represent categorical features. In other words each categorical feature is transformed such that each category (or value) in the feature is transformed to the probability of that category occurring in the feature. Hence the column now represents a probability distribution of the categories in the categorical feature.

My question: Does it make sense to transform categorical features in this way? My impression is that since I am using a neural network this approach is appropriate, but for distance or density based algorithms this approach would not be suitable as probability distributions shouldn't be evaluated spatially (as with a distance or density metric).

I have read the below posts, but they do not discuss using a probability distribution.

Categorical Features for NN

Mixed features for NN

Answer 1

As long as the frequency of the categories isn´t correlated with the target, it shouldn´t make any sense.

The paper you mentioned is solely targeted for anomaly detection where it is obvious that the frequency and target variable (Outlier Y/N) may be connected.

In an extreme example, you have a variable with two distinct values that have an equal amount of values. Therefore the explanatory power is null.

I would rather use Mean Encoding for categorical variables. But be aware of overfitting.

Answer 2

I have found this article that discusses an alternate method for encoding variables in the supervised case. Since it demonstrates the use of frequency encoding for supervised learning, the same encoding would also be feasible for the unsupervised case. The authors also present their own (superior) method to encoding variables.