I have a large dataset of consumer loans. It is not, strictly speaking, time series data, but as customers take repeated loans over time at fairly regular intervals, there is a very strong chronological component. Fields are mostly engineered fields describing a customer's past loan performance, as well as boolean fields for attributes such as the loan officer, regional location, month, etc. The data set spans 4 years.
I'm using my data to train a random forest model to predict loan defaults. If I do an 80/20 train-test split uniformly throughout the data, I get reasonable numbers (ROC-AUC around 0.75). If I do an 80/20 split chronologically (training on the first 80% of the data and testing on the next 20%, through the time dimension), however, with the rest of the model unchanged, the ROC-AUC drops into the low 50s. Aside from the chronological train-test split the model is otherwise unchanged. The proportion of defaults in the testing set is fairly constant, whether the train-test split is made chronologically or uniformly.
My worry is that the model is overfitting to certain combinations of variables (like dummies for month, location, etc.) which correlate to historical patterns (i.e. specific instances of localized fraud) which shouldn't actually be included in a forward-looking predictive model. Most of what I've read on Random Forests discourages artificially shortening trees or limiting splits, but I'm wondering if there are any specific techniques for avoiding overfitting when doing chronological splits? Conceptually I've been thinking about ideas like walk-forward k-fold validation to tune feature selection, but haven't found any literature on how to tackle this kind of feature selection outside of randomly pulling features in/out of the model. (Most of what I've looked at discourages feature selection in Random Forests, and instead just recommends building more trees). Any advice on specific techniques for chronological overfitting would be really appreciated.