I have a Pandas dataframe for which I've performed some actions over categorical features:
- Feature Engineering
- One-Hot Encoding
Let's say that in my dataset I have the features "person_income" and "address_neighborhood".
For address_neighborhood, I want to generate a new feature which will be "neighborhood_average_income", which will be simply the average income for people residing in that particular neighborhood.
I'm doing that with Pandas by executing the following:
average_income = df.groupby('address_neighborhood').mean().person_income df['neighborhood_average_income'] = df.address_neighborhood.apply(lambda x: average_income.loc[x])
After creating a new, numerical variable based off my categorical variable, I do some one-hot encoding based on it, which results in a range of new features.
I'm aware that doing the feature engineering steps described above over the full dataset and then splitting into training and testing sets might lead to data leakage, as I would be exposing the test set to aggregated values of the full dataset.
But doing the steps above for both training and test sets would cause me a problem as well, as I could get a different number of dummy variables resulting from one-hot encoding, and thus leading to different number of columns in the training and testing sets.
Is there a better way to deal with this problem?