I am working on a classification problem and have engineered a few categorical features with high cardinality by dummying out the most frequently occuring values and then using the response variable to bucket a group of less frequently occuring values with high positive response rates and dummying out this group as its own category.
I am ready to run a few base models and evaluate their performance but am wondering how I will cross-validate since splitting my training set into separate folds and holdout sets would result in data leakage due to the method I used for bucketing values with high positive response rates. I'm wondering if I should instead split my test set into separate folds (5 folds) and take the average score after testing on a different combination of test folds (hold out one fold from each test)?
I would appreciate any feedback or suggestions on how to proceed in such a way that will avoid any data leakage but still allow me to perform cross-validation and avoid overfitting.