Currently, I am working on a binary classification project that include both numeric and categorical variables as predictors. I recently read an article about encoding variable with weight of evidence in
R - http://www.r-bloggers.com/r-credit-scoring-woe-information-value-in-woe-package/. I encoded my categorical variables and also two way interactions of the categorical variables with weight of evidence. I used a 10-fold cross validation and the encoding performed really well on my training set but dismal on my test set. The model was really overfit.
My question is -
- When doing such encodings what is the best practice to get the good performance of such transformations while reducing overfitting?
My training set has about 114k records and I am using
xgboost as my classification algorithm.