Let's say I have user data (x) for customers (i) in the period between 2010 and 2020 (t). I want to predict if customer i churns at time t. To do so, I have built a stacked model which looks somewhat like:

Client  |     Date    |  Logins |   Churn
   1       2010-01-01      20         0
   1       2010-02-01      25         0
  ...          ...         ...       ...
  100      2019-01-01      3          0
  100      2019-02-01      6          1

Two questions come to mind after looking at the data (including more variables) before running my XGBoost model:

  • What if there is an overall trend in time with regard to the number of logins for all users. Should we replace a detrended number of logins instead?
  • There a relative big difference between the number of logins per user. Should the number of logins also be detrended on a client level so that the effect of a decrease in the number of logins per client also contributes to the churn probability?

To some extent, it depends on how well each of these ideas performs. Of course, it's pretty expensive and time consuming to do predictions with the different approaches, wait some months and then find out whether a model worked. And this gets even messier, when you aim to intervene based on the predictions. So, the next best thing is to cut your data in time: e.g. predict based on everything we knew before 1 January 2015 vs. what happened in 2015, predict based on everything we knew before 17 April 2016 vs. what happened in 2016, or whatever corresponding splits make sense. Ideally, you would want to see that are model performs consistently across cuts (plus minus a bit of noise and taking into account that the later you do the cut, the more training data you have - unless you actually want to discard very old training data, which may be a serious consideration).

As you suggest, I would expect that taking into account timeframes will matter. E.g. number of logins a user ever made is likely not that great a predictor, because it confounds two different types of information: how much the user logs in and how long they have been around. So, instead things like number of log-ins per year of being a user, log-ins in the last 2 week, last 12 weeks, last half-year etc. are alternatives that sound like things one should investigate (and there are probably other ways of creating features of this type). It also sounds plausible to look at the relative change in recent (whatever that means e.g. last month?) number of logins vs. how many the user usually (whatever that means) does.

In principle, a model like xgboost is capable of figuring out interactions (e.g. when a user is on the platform for a long time, a large number of logins means something different than for a user that's been there a short-time), but it tends to be much, much more efficient to provide a feature that captures a human understanding. Additionally, xgboost cannot really extrapolate, unless you give it features that help it extrapolate: What I mean is that the training data has no user with >10 years membership and no user with >1000 logins, then when get new data that has 15 years membership and 15000 logins, it will actually treat it just like 10 years membership with 1000 logins - in contrast, there might be existing users with >1000 logins/year (assuming that this is a sensible way of looking at the data), which the model could base predictions on (we can argue whether the model should do that and you need to think about that).

  • $\begingroup$ Thanks @Björn for the extensive answer, very helpful. I do have two more questions based on your answer: 1) Why splitting the data per year and analyzing it separately, isn't adding a dummy for year enough? 2) If we have many users (read 100k+), we don't use user as a variable right? So how does the model figure out the trend within one user? $\endgroup$
    – Michieldo
    Nov 25 '20 at 14:30
  • $\begingroup$ With splitting by time I meant taking some date, use data before to predict what comes after (could focus on years, but does not have to). That's the logical approach to check a model actually works for data like this with a time series nature. This blog post (and probably other places) has a good discussion of this topic: fast.ai/2017/11/13/validation-sets (see "Time Series" Section). $\endgroup$
    – Björn
    Nov 26 '20 at 11:29
  • 1
    $\begingroup$ Using user id is an option, but you don't want your model to learn that existing users are immune to churn (because they are still users), while those that no longer are, are predicted to churn. You want it to learn what it is about a user that makes them churn rather than "I predict this because it's user 7235691 - can't tell you more". Using user id probably makes the most sense in a setting were users churn, but then come back later (and you can link the records). However, even then it may be simpler to capture "everything" about a user in features (e.g. no. of previous churns). $\endgroup$
    – Björn
    Nov 26 '20 at 11:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.