I'm working on evaluating various classification algorithms to help predict customer churn (or at least ID interesting features to use in later strategy).

The goal is to identify accounts who are at risk for not buying again. As a first step, I took:

a) all customers who were active between 250 and 120 days ago (somewhat arbitrary)
b) classified them as churned or not churned based on whether they were active in the last 110 days.

One of my features is "days since last logged in". In order to purchase, you need to log-in. But, you may log-in to do a variety of things that have nothing to do with ordering (for instance, account mgmt, download receipts, etc....).

Since there's a clear relationship between "not churned" and logging in, what are your thoughts on how valuable this feature is? So far, my arguments for including it:

It's valid and accessible real-time information. If a customer has not purchased in 5 months and we saw they logged in yesterday, why offer a discount or take action to re-engage the client?

Has anyone dealt with similar cases where the feature is somewhat dependent on the labeling criteria?

  • $\begingroup$ "What are your thoughts" is not a great fit for CV. Can you make your question more focused & concrete? $\endgroup$ – gung - Reinstate Monica Mar 26 '16 at 19:20
  • $\begingroup$ @gung - did you read the question? I provided a concrete use case and was looking for specific advice regarding the treatment of a potentially useful feature in a classification model. How is this a bad fit for CV? $\endgroup$ – jrjames83 Mar 26 '16 at 19:28
  • $\begingroup$ I did read the question, & I did not downvote or vote to close. It's just the "what are your thoughts on how valuable this feature is" that is potentially problematic. You may want to edit that. $\endgroup$ – gung - Reinstate Monica Mar 26 '16 at 19:54
  • $\begingroup$ I would suggest that "collinearity" may not be the correct word to use in the title. Collinearity refers to dependence among the predictors that can degrade model fit insofar as it will inflate the standard errors of the parameter estimates. A better term might be "endogeneity" which reflects undesirable dependence between predictors and a target or dependent variable. That said, and as @Vishal notes, in this case, it's neither a theoretical nor applied issue. $\endgroup$ – Mike Hunter Dec 30 '16 at 15:59

It’s perfectly okay to use “days since last log-in” as one of the independent variables in the churn model. This independent variable does not, in and of itself, define churn. And it’s obvious that if a customer has not logged on for a long time (i.e., lagged) then she is more likely to churn in the near future.

The important distinction here is that the “days since last log-in” is extracted from a time frame that is mutually exclusive – and it precedes – the model observation window from which the outcome event (churn) is defined. Since there’s no overlap between the two, “days since last log-in” is truly a leading indicator, and hence it should be okay to use this, or any other such feature, for that matter.

This is a common practice in such models that use Recency/Frequency/Monetary (RFM) variables to make a prediction.

By the way, in the past, when I built churn models (and I have built many) using such ‘recency’ fields, one of the issues that I ran into is the difficulty of building a strong churn model that can satisfactorily outperform a baseline churn “model” that simply ranks order customers based on recency.

If we simply rank order all customers by their recency of purchase, this “model” would be able to capture the churners in the top decile very efficiently (and parsimoniously). So the real test of the churn model is not how well it performs against a random ranking (i.e., AUC), but how well it can outperform the single-variable (recency) model.

Another important factor is the definition of churn itself. You mentioned that the choice of 250—120 days was arbitrary. However, you might want to consider defining a specific window of inactivity as churn. You can look at periods of inactivity from historical data to determine this window. For instance, you might find something like this: “95% of the customers who stayed inactive for 40 days did not make any subsequent purchase” – based on this, you may define churn as 40 consecutive days of inactivity. (The business point of view might add some more nuances/justification to this.) Once you define churn this way, you can select the historical window more systematically, instead of the arbitrary choice of 250—120 days.

  • $\begingroup$ Awesome feedback - thinking this over. I had the same intuition that a grid which just ranks customers by LTV and recency, is likely the measure to beat, at least in practical terms. $\endgroup$ – jrjames83 Mar 30 '16 at 16:53
  • $\begingroup$ Glad you found it useful. Would you mind accepting my answer then? $\endgroup$ – Vishal Mar 30 '16 at 17:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.