LabelEncoder with a Multi-Layer Perceptron?

Question

So we're working on a machine learning project at work and it's the first time I'm working with an actual team on this. I got pretty good results with a model that uses the following SKLearn pipeline:

Data -> LabelEncoder -> MinMaxScaler (between 0-1) -> PCA (I go from 130 columns to 50 prime components that cover the variance) -> MLPRegressor

One of my colleagues mentioned that I shouldn't normally use LabelEncoder to encode training data, as it's meant for encoding the target variable. I did some research and now and I understand why

LabelEncoder only is not a good choice, since it brings in a natural ordering for different classes.

Then, however, my colleague mentioned that in this case it shouldn't make much of a difference as I'm using a neural net (~MLPRegressor). My question is - (if he's right - is he?) why? He basically commended me saying this usually would be a bad idea but in this case it should work.

I will try to move to one-hot encoding (currently I'm only stuck with it because I run out of memory while doing PCA on that many columns, but that's another question and I'll do some research on that separately), but for now I'd like to know if using this kind of encoding can result in inaccurately good results (I'm having an r^2 score of around 0.9 and my boss literally won't believe I achieved a result that good haha).

Answer 1

No, independent of the model you use, it's not a good idea. It's not the case that MLP handles or understands it. Imposing an ordinality may create unexpected problems. Your model performance is probably good because of another feature subset.

Answer 2

Well, you can use a label encoder on training data, but I typically don't. If there are just two categories, you could use it safely (I just tend to apply a function that recodes the data to 0s and 1s.) However, if there are more than two categories and and the categories don't represent an ordinal scale like tickets (tickets could be first-class, second-class, third-class), then you should probably one-hot encode, just as the article you linked to suggested. If the values truly represent ordinal data, one can use an OrdinalEncoder.

In terms of the MLPRegressor, if you run a label encoder on a multi-value categorical column that's been label encoded, and the categories don't represent ordinal data, the NN has no idea of knowing that the values represented aren't meaningful in terms of magnitude. The good results you're getting are in spite of the use of the label encoder.