# One-Hot Encoding and Feature Engineering While Avoiding Data Leakage

I have a Pandas dataframe for which I've performed some actions over categorical features:

1. Feature Engineering
2. One-Hot Encoding

Let's say that in my dataset I have the features "person_income" and "address_neighborhood".

Feature Engineering

For address_neighborhood, I want to generate a new feature which will be "neighborhood_average_income", which will be simply the average income for people residing in that particular neighborhood.

I'm doing that with Pandas by executing the following:

average_income = df.groupby('address_neighborhood').mean().person_income


One-Hot Encoding

After creating a new, numerical variable based off my categorical variable, I do some one-hot encoding based on it, which results in a range of new features.

Problem

I'm aware that doing the feature engineering steps described above over the full dataset and then splitting into training and testing sets might lead to data leakage, as I would be exposing the test set to aggregated values of the full dataset.

But doing the steps above for both training and test sets would cause me a problem as well, as I could get a different number of dummy variables resulting from one-hot encoding, and thus leading to different number of columns in the training and testing sets.

Is there a better way to deal with this problem?

Define the one-hot encoding bucket definitions based on the training set. Use the same bucket definitions for the test set.

Example: your training set has: 1,2,3,1,2,1,5,10

You decide to create buckets:

• 0
• (0 - 2]
• (2 - 5]
• (5 - 10]
• (more than 11]

You can use these buckets for both training and test set, with no further modifications needed for the test set, because the buckets are both mutually disjoint, and cover all possible values.