# One hot encoding vs apply the average of the label to each category

I have a fairly reasonably sized dataset (row>50k). And I'm looking for the best way to utilize some of the categorical columns. For purpose of this question, let's say that one of the categorical column is zipcode. The premise is, after feature engineering, I'll pass this data to a random forest regressor in sklearn, which does not recognize categorical columns.

Let's say I have 500 unique zipcode. I could one-hot encode these, or pick the top 100 and then one-hot encode those (fill the rest with "OTHER" for instance), but they all generate a large amount of dimensionality, which I wanted to avoid. Here's the new idea that I have and I want to verify it with the community.

Let's say after I do train-test split, I take the train set, and average the individual zipcode group by the real labels they have, so for instance in the raw data:

zipcode  label
zip10001 3
zip10001 2
zip10001 4
zip10002 1
zip10002 2
zip10010 7


after transform, becomes

zipcode  label zipcode_avg
zip10001 3     3
zip10001 2     3
zip10001 4     3
zip10002 1     1.5
zip10002 2     1.5
zip10010 7     7


while also creating a dictionary:

dzipavg = {
"zip10001": 3,
"zip10002": 1.5,
"zip10010": 7
}


And instead of one-hot encoding the zipcode column, I'll simply drop it. And for the test column, I would map the zipcode with the dict test.["zipcode_avg"] = test.zipcode.map(dzipavg), and drop the zipcode column as well.

Do you think this is a good idea? Will there be any consequences that I have not seen? I don't think there's any data leak in here as all transformation is based on training data.

• This method does in fact lead to data leakage and overfitting (high bias), because you are using target/label data to create features for your model. This can be addressed with regularization. Apr 26, 2019 at 22:04
• @AlexK I can't see how this lead to data leakage, no information is taken from the testing set of the split. Apr 26, 2019 at 22:14
• Data leakage in broad terms is defined as this: "if any feature whose value would not actually be available in practice at the time you'd want to use the model to make a prediction, it is a feature that can introduce leakage to your model." (dataskeptic.com/blog/episodes/2016/leakage) This target-based feature, in your case, would fall under that definition because you would not have target values at the time of testing. Apr 26, 2019 at 22:23
• @AlexK That's not true though, the whole purpose of the train test split is to assume we already have the information of the training set, which contains the target. If target(label) is not available, there would be nothing to train on in supervised learning model. Apr 27, 2019 at 23:48
• Look up what data leakage is. I am not talking about not using target values to train the model. I am talking about creating features that are based on target values. The model becomes more accurate than it should on the training data as a result. Apr 28, 2019 at 2:32