I've attempted to tackle the following problem: When a customer has registered on our site, we'd like to predict if he/she will eventually leave us. I have 3 days' worth of data, that captures behaviors that he/she has done on the site for the first 3 days since registration. The data look like below: (100+ additional features omitted)

Customer ID Capture_date X1 X2 X3 X4 Event_occurred Event_occurred_date Target
1 1/3/2020 null 2 N 8.75 1 3/15/2020 1
1 1/4/2020 null 4 Y 8.75 1 3/15/2020 1
1 1/5/2020 3 4 Y 8.75 1 3/15/2020 1
2 3/31/2020 null 0 N 0 1 4/1/2020 1
3 2/4/2020 22 0 Y 13.64 0 null 0
3 2/5/2020 22 16 Y 22.08 0 null 0

For each customer ID, at most I'll have 3 rows. ID 3 only has 2 rows because 2/6 records were the same as 2/5. ID 2 has 1 row as he/she left on 4/1, and ID 1 has all 3 records because there are changes on any of the 4 features.

My question is: how should I model this? In production, the trained model will be run every day on new customers for up to 3 days. Once a customer is flagged as "likely to leave", we'll trigger a set of actions to save him/her. However, if someone is flagged as "not likely to leave" on day 1, he/she would be subject to rescoring the next day if any of the features have updated values.

My thoughts:

  • This could be framed as a survival analysis problem, and I've looked into the lifelines package. However, I don't think the time-varying Cox model exposes an interface on predicting survival functions. Plus, I feel that survival analysis is more useful for inferential analysis, not predictive modeling. I'm new to this field, so feel free to correct me if I'm wrong.
  • My next thing to try is random effects, but with the fixed target, I wonder how much it could help. Plus, the training dataset is huge - we have 100K+ customer records, I'm worried that using Customer ID as a random effect may introduce too many levels. The goal is not too relevant about retaining old customers that are already in the training data; rather, the business is interested in scoring new customers.
  • I've tried to train classification models directly on the raw training data by creating a time lag variable. So far, the performance is far from expected. I've carefully split the train/test data so that every unique customer only occurs on either train or test. However, correlations among samples cannot be mitigated.

Should I just use the first/last record per customer to train an ML model instead? I think the business would like to discover if a customer is going to fall off ASAP. However, if using day 1 data only it could be premature. I just can't get wrap my mind around having this model run on every active customer every day since registration for up to 3 days. Any help is appreciated!!!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.