I am new to machine learning. I am working on churn prediction for a customer. I am wondering how best to partition the data for training/test/production deployment. My thinking is that churning customers are going indicate their intention to churn by how they use the SaaS product. This could include reduced usage of some features, or just a drop overall in product usage. There could be other features too that indicate early churn such as reduced/increased calls into support etc. Anyhow, I am facing a problem on how to partition my data set for train/test and then eventually for how to handle production deployment. The data is in CSV format with each row representing a customer and each column representing a feature. For example, a column could be how many work orders created up to the point in time of the export i.e. a cumulative export of total count of work orders up to the time of creating the export.
My options for creating the data export are:
- A. Each week, take an export of all customers and all activity in that week, or,
- B. Each week, take an export of all customers and all cumulative activity up to that week from the point when they became a customer.
My options for train/test are:
-
- Train logistic regression model on 80% of weekly export from last week (either option A/B above) and test against 20%. Then apply model to this weeks data for all 100% of customers in production to see which ones churned, or,
-
- Train logistic regression model on 100% of data from 2 weeks ago and test on last weeks data. Then apply model to this weeks data in production.
Any thoughts on both creating the data export as well as how to partition the data for train/test?
