1
$\begingroup$

I have a Pandas dataframe for which I've performed some actions over categorical features:

  1. Feature Engineering
  2. One-Hot Encoding

Let's say that in my dataset I have the features "person_income" and "address_neighborhood".

Feature Engineering

For address_neighborhood, I want to generate a new feature which will be "neighborhood_average_income", which will be simply the average income for people residing in that particular neighborhood.

I'm doing that with Pandas by executing the following:

average_income = df.groupby('address_neighborhood').mean().person_income
df['neighborhood_average_income'] = df.address_neighborhood.apply(lambda x: average_income.loc[x])

One-Hot Encoding

After creating a new, numerical variable based off my categorical variable, I do some one-hot encoding based on it, which results in a range of new features.

Problem

I'm aware that doing the feature engineering steps described above over the full dataset and then splitting into training and testing sets might lead to data leakage, as I would be exposing the test set to aggregated values of the full dataset.

But doing the steps above for both training and test sets would cause me a problem as well, as I could get a different number of dummy variables resulting from one-hot encoding, and thus leading to different number of columns in the training and testing sets.

Is there a better way to deal with this problem?

$\endgroup$
1
$\begingroup$

Define the one-hot encoding bucket definitions based on the training set. Use the same bucket definitions for the test set.

Example: your training set has: 1,2,3,1,2,1,5,10

You decide to create buckets:

  • 0
  • (0 - 2]
  • (2 - 5]
  • (5 - 10]
  • (more than 11]

You can use these buckets for both training and test set, with no further modifications needed for the test set, because the buckets are both mutually disjoint, and cover all possible values.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.