So this is more of a theoretical question, no dataset or code that I can share. It just came up in a discussion and I was not sure of the answer.

Let's say I have 2 GBM models, model A and B, trained on the same training set, predicting some 0/1 dependent variable. And let's say we have a test set in which we can assess the performance of these two models. Both training and test sets came from the same time period of data collection, just a random split

Now let's assume Model A is much less complicated than model B, maybe 100 trees vs 1000 trees, but the performance gains are minimal, maybe 70 vs 71 AUC on the test set.

My question is, which model here is more sensitive to changes in variable distributions over time? For example, if we are using these 2 models in real time decisions, and a mean of a variable that we use in both models shifts, which model is more likely to be impacted by this?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.