I am looking for the best cross-validation strategy to test the performance of a churn prediction machine learning model (classification).

The model predicts if a client is going to churn in the next 6 months. Each month we have usage data for each customer. So it is basically a time series dataset per client. The data is highly imbalanced like a lot of churn datasets are where the churn even is pretty rare.

I tried two cross-validation strategy. First a StratifiedGroupKFold cross-validation strategy where each fold doesn't contain the same clients and each fold contain the same class distribution of churn vs non-churn. This basically ensures that the model is tested on clients it never seen before in the training data, preventing data leakage. But it tests the model on data in the past and the future... This cross-validation strategy doesn't take time into account.

For the second cross-validation strategy I simply split by time and try to predict observations in the future.

The model gets a good AUC score when using StratifiedGroupKFold, but much worse when splitting by time.

Not too sure how to interpret that. The model is not good at extrapolating but he can predict client it never seen before pretty good?


  • 1
    $\begingroup$ Have you had a look at medium.com/@soumyachess1496/…. $\endgroup$
    – discipulus
    May 25, 2021 at 6:44
  • $\begingroup$ @discipulus thank you for the link the info was really relevant. This seems to be plagiarism from an earlier article though. Including the link to give credit to the original author: towardsdatascience.com/… $\endgroup$
    – EtienneT
    May 28, 2021 at 18:16
  • 1
    $\begingroup$ Yes! It seems so. There has been a comment in the article itself; it certainly isn't mine. Thanks! I know which one I should refer to. $\endgroup$
    – discipulus
    May 31, 2021 at 6:00


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.