Avoiding Data Leakage from Bucketed Features During Cross-Validation

Question

I am working on a classification problem and have engineered a few categorical features with high cardinality by dummying out the most frequently occuring values and then using the response variable to bucket a group of less frequently occuring values with high positive response rates and dummying out this group as its own category.

I am ready to run a few base models and evaluate their performance but am wondering how I will cross-validate since splitting my training set into separate folds and holdout sets would result in data leakage due to the method I used for bucketing values with high positive response rates. I'm wondering if I should instead split my test set into separate folds (5 folds) and take the average score after testing on a different combination of test folds (hold out one fold from each test)?

I would appreciate any feedback or suggestions on how to proceed in such a way that will avoid any data leakage but still allow me to perform cross-validation and avoid overfitting.

Answer 1

This is a good question. I think what you are describing is called "impact coding". Impact coding is like a tiny model, and so it needs to be "trained" on its own data set. It is advised on that blog and elswhere that there should be 3 data sets.

The first is used to create the impact codings. For the reasons you mention, this data can not be used to learn a model due to leakage.

The second is your training set. You apply the impact codes you learn in the previous step to this data. I think the blog I've linked to might have some code to make this easy.

The third is the test set.