# How to prevent overfitting when encoding categorical variables

Currently, I am working on a binary classification project that include both numeric and categorical variables as predictors. I recently read an article about encoding variable with weight of evidence in R - http://www.r-bloggers.com/r-credit-scoring-woe-information-value-in-woe-package/. I encoded my categorical variables and also two way interactions of the categorical variables with weight of evidence. I used a 10-fold cross validation and the encoding performed really well on my training set but dismal on my test set. The model was really overfit.

My question is -

1. When doing such encodings what is the best practice to get the good performance of such transformations while reducing overfitting?

Note: My training set has about 114k records and I am using xgboost as my classification algorithm.

In order to avoid overfitting while using WOE transformations, you should use 'penalty' to discard transformed features that don't hold up against the validation set. Check out the 'information' package: https://cran.r-project.org/web/packages/Information/Information.pdf

Also, here's a more elaborate explanation and demonstration of how 'penalty' should be used: http://multithreaded.stitchfix.com/blog/2015/08/13/weight-of-evidence/