Prudent to reduce data size for the sake of model performance?

Question

I am currently working on predicting the customer revenue in next 3,6 or 9 months using the below two methods

a) Buy Till you die probabilistic models b) Tweedie regression and other regression techniques

Currently, I have data from 2017-01-01 to 2022-05-30 of 5900 customers.

However, out of this 5900 customers, only 1500-2000 are active or loyal from the beginning.

While I tried building models using all 5 year data for all customers (with train and test), I don't see good model performance.

However, when I chose to model only using last 2 years data (for whichever customer had data), I do see a quick jump in my metrics (R2 for regression, MAE, RMSE etc)

So, is it advisable to reduce the dataset size for the sake of model performance?

Or are we losing any info by reducing dataset size (and is it a bad practice?)

Answer 1

This sounds like a standard case of model drift, in other words, the relationship between your predictors and your outcome change over time so that a model trained on old data might not make good predictions today.

When this happens, you have to be careful in how you split your data into train, validation, and test sets. For instance, you probably would want to use only the most recent data as the test set, and ensure that your validation data always comes from a later date than your training data. You can find more on this in any article on model drift.