I've attempted to tackle the following problem: When a customer has registered on our site, we'd like to predict if he/she will eventually leave us. I have 3 days' worth of data, that captures behaviors that he/she has done on the site for the first 3 days since registration. The data look like below: (100+ additional features omitted)
For each customer ID, at most I'll have 3 rows. ID 3 only has 2 rows because 2/6 records were the same as 2/5. ID 2 has 1 row as he/she left on 4/1, and ID 1 has all 3 records because there are changes on any of the 4 features.
My question is: how should I model this? In production, the trained model will be run every day on new customers for up to 3 days. Once a customer is flagged as "likely to leave", we'll trigger a set of actions to save him/her. However, if someone is flagged as "not likely to leave" on day 1, he/she would be subject to rescoring the next day if any of the features have updated values.
- This could be framed as a survival analysis problem, and I've looked into the
lifelinespackage. However, I don't think the time-varying Cox model exposes an interface on predicting survival functions. Plus, I feel that survival analysis is more useful for inferential analysis, not predictive modeling. I'm new to this field, so feel free to correct me if I'm wrong.
- My next thing to try is random effects, but with the fixed target, I wonder how much it could help. Plus, the training dataset is huge - we have 100K+ customer records, I'm worried that using
Customer IDas a random effect may introduce too many levels. The goal is not too relevant about retaining old customers that are already in the training data; rather, the business is interested in scoring new customers.
- I've tried to train classification models directly on the raw training data by creating a time lag variable. So far, the performance is far from expected. I've carefully split the train/test data so that every unique customer only occurs on either train or test. However, correlations among samples cannot be mitigated.
Should I just use the first/last record per customer to train an ML model instead? I think the business would like to discover if a customer is going to fall off ASAP. However, if using day 1 data only it could be premature. I just can't get wrap my mind around having this model run on every active customer every day since registration for up to 3 days. Any help is appreciated!!!