1
$\begingroup$

Let's say we are trying to train a propensity model that predicts churn or conversion (the probability to stop using a service or subscribe). For both models, we have a dilemma. On the one hand, for testing purposes, we would like to have the most recent data for the holdout dataset to better understand the model's performance in the wild. On the other, by doing so, we are excluding an important and recent part of the data from the training procedure, which doesn't seem a great idea either.

In general, the features used are quite standard and include information about users and their behavior.

What would you suggest as the most sensible approach to validation for such tasks?

Improve this question
$\endgroup$
3
  • $\begingroup$ Yeah definitely an important thing to consider which is left out a lot in standard time series validation. You could do a test for large structural changes that appear in your holdout and then potentially truncate your holdout to make sure that it is taken into account. Just make sure to keep enough data points in your validation! $\endgroup$
    – Tylerr
    Commented Apr 26, 2021 at 18:57
  • $\begingroup$ @Don Draper - I am also faced with same doubt (amd deal with similar problem) May I know what did you do and can also share your thought process behind it? $\endgroup$
    – The Great
    Commented Mar 27, 2022 at 13:42
  • $\begingroup$ @Tylerr - can I have your views on this relatef post as well? stats.stackexchange.com/questions/569295/… $\endgroup$
    – The Great
    Commented Mar 27, 2022 at 13:44

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.