0
$\begingroup$

I am working on a classification problem and have engineered a few categorical features with high cardinality by dummying out the most frequently occuring values and then using the response variable to bucket a group of less frequently occuring values with high positive response rates and dummying out this group as its own category.

I am ready to run a few base models and evaluate their performance but am wondering how I will cross-validate since splitting my training set into separate folds and holdout sets would result in data leakage due to the method I used for bucketing values with high positive response rates. I'm wondering if I should instead split my test set into separate folds (5 folds) and take the average score after testing on a different combination of test folds (hold out one fold from each test)?

I would appreciate any feedback or suggestions on how to proceed in such a way that will avoid any data leakage but still allow me to perform cross-validation and avoid overfitting.

$\endgroup$
1
$\begingroup$

This is a good question. I think what you are describing is called "impact coding". Impact coding is like a tiny model, and so it needs to be "trained" on its own data set. It is advised on that blog and elswhere that there should be 3 data sets.

The first is used to create the impact codings. For the reasons you mention, this data can not be used to learn a model due to leakage.

The second is your training set. You apply the impact codes you learn in the previous step to this data. I think the blog I've linked to might have some code to make this easy.

The third is the test set.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.