0
$\begingroup$

I've attempted to tackle the following problem: When a customer has registered on our site, we'd like to predict if he/she will eventually leave us. I have 3 days' worth of data, that captures behaviors that he/she has done on the site for the first 3 days since registration. The data look like below: (100+ additional features omitted)

Customer ID Capture_date X1 X2 X3 X4 Event_occurred Event_occurred_date Target
1 1/3/2020 null 2 N 8.75 1 3/15/2020 1
1 1/4/2020 null 4 Y 8.75 1 3/15/2020 1
1 1/5/2020 3 4 Y 8.75 1 3/15/2020 1
2 3/31/2020 null 0 N 0 1 4/1/2020 1
3 2/4/2020 22 0 Y 13.64 0 null 0
3 2/5/2020 22 16 Y 22.08 0 null 0

For each customer ID, at most I'll have 3 rows. ID 3 only has 2 rows because 2/6 records were the same as 2/5. ID 2 has 1 row as he/she left on 4/1, and ID 1 has all 3 records because there are changes on any of the 4 features.

My question is: how should I model this? In production, the trained model will be run every day on new customers for up to 3 days. Once a customer is flagged as "likely to leave", we'll trigger a set of actions to save him/her. However, if someone is flagged as "not likely to leave" on day 1, he/she would be subject to rescoring the next day if any of the features have updated values.

My thoughts:

  • This could be framed as a survival analysis problem, and I've looked into the lifelines package. However, I don't think the time-varying Cox model exposes an interface on predicting survival functions. Plus, I feel that survival analysis is more useful for inferential analysis, not predictive modeling. I'm new to this field, so feel free to correct me if I'm wrong.
  • My next thing to try is random effects, but with the fixed target, I wonder how much it could help. Plus, the training dataset is huge - we have 100K+ customer records, I'm worried that using Customer ID as a random effect may introduce too many levels. The goal is not too relevant about retaining old customers that are already in the training data; rather, the business is interested in scoring new customers.
  • I've tried to train classification models directly on the raw training data by creating a time lag variable. So far, the performance is far from expected. I've carefully split the train/test data so that every unique customer only occurs on either train or test. However, correlations among samples cannot be mitigated.

Should I just use the first/last record per customer to train an ML model instead? I think the business would like to discover if a customer is going to fall off ASAP. However, if using day 1 data only it could be premature. I just can't get wrap my mind around having this model run on every active customer every day since registration for up to 3 days. Any help is appreciated!!!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.