Let's say we are trying to train a propensity model that predicts churn or conversion (the probability to stop using a service or subscribe). For both models, we have a dilemma. On the one hand, for testing purposes, we would like to have the most recent data for the holdout dataset to better understand the model's performance in the wild. On the other, by doing so, we are excluding an important and recent part of the data from the training procedure, which doesn't seem a great idea either.

In general, the features used are quite standard and include information about users and their behavior.

What would you suggest as the most sensible approach to validation for such tasks?

  • $\begingroup$ Yeah definitely an important thing to consider which is left out a lot in standard time series validation. You could do a test for large structural changes that appear in your holdout and then potentially truncate your holdout to make sure that it is taken into account. Just make sure to keep enough data points in your validation! $\endgroup$
    – Tylerr
    Apr 26 at 18:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.