How to Find Out Which Features Are Contributing to Overfitting on Random Forest

Question

I have a large dataset of consumer loans. It is not, strictly speaking, time series data, but as customers take repeated loans over time at fairly regular intervals, there is a very strong chronological component. Fields are mostly engineered fields describing a customer's past loan performance, as well as boolean fields for attributes such as the loan officer, regional location, month, etc. The data set spans 4 years.

I'm using my data to train a random forest model to predict loan defaults. If I do an 80/20 train-test split uniformly throughout the data, I get reasonable numbers (ROC-AUC around 0.75). If I do an 80/20 split chronologically (training on the first 80% of the data and testing on the next 20%, through the time dimension), however, with the rest of the model unchanged, the ROC-AUC drops into the low 50s. Aside from the chronological train-test split the model is otherwise unchanged. The proportion of defaults in the testing set is fairly constant, whether the train-test split is made chronologically or uniformly.

My worry is that the model is overfitting to certain combinations of variables (like dummies for month, location, etc.) which correlate to historical patterns (i.e. specific instances of localized fraud) which shouldn't actually be included in a forward-looking predictive model. Most of what I've read on Random Forests discourages artificially shortening trees or limiting splits, but I'm wondering if there are any specific techniques for avoiding overfitting when doing chronological splits? Conceptually I've been thinking about ideas like walk-forward k-fold validation to tune feature selection, but haven't found any literature on how to tackle this kind of feature selection outside of randomly pulling features in/out of the model. (Most of what I've looked at discourages feature selection in Random Forests, and instead just recommends building more trees). Any advice on specific techniques for chronological overfitting would be really appreciated.

Answer 1

In this case, the design itself – and not just certain variables – contributes to overfitting. Independence is still an important consideration in RF, and you are referring to strong non-independence. So, specific advice on chronological ML: RE-EM; repeated-measures ; and more clustered data.

Secondly, there are pre-analysis common-sense approaches: if you know that some ‘variables should not actually be included’, they need to be eliminated a priori. This is not the same as ‘feature selection’ but eliminating junk variables which would not be helpful in the forward model, based on the information you provide.

Aside from that, there are internal approaches to account for some of the overfitting, such as this beautiful simulation of mtry greediness as well as other fine-tuning methods.