Churn Model Performance Measurement Metrics & KPI For Business

Question

I have a customer churn model, which classifies people who are going to leave (Yes) from those who are staying (No). I trained my model using 10-fold Cross-validation and I used AUC under ROC curver as a metric for my model performance, to begin with.

However, I went through many nice readings on the subject and I got difference at times confusing suggestions. Which can be summarised as follow.

• Accuracy will not always be the metric.
• Precision and recall are often in tension. That is, improving precision typically reduces recall and vice versa.
• AUC-ROC curve is one of the most commonly used metrics to evaluate the performance of machine learning algorithms.
• ROC Curves summarise the trade-off between the true positive rate and false positive rate for a predictive model using different probability thresholds.
• The ROC curve can be used to choose the best operating point.

$Question:$ Which metric I should I give to business telling them my model is doing a good job

$Question:$ I need to give some number of business KPI; which should be interpretable to them. What could be my choice to business?

DEPLOYED MODEL PERFORMANCE

01 - January

## CONFUSION TABLE
> estimates_rf_tbl %>% conf_mat(truth, estimate)
          Truth
Prediction     No    Yes
       No  229493   6604
       Yes  34687  10497

## OVER MODEL PERFORMANCE
> tibble(
+     auc             = estimates_rf_tbl %>% roc_auc(truth, class_prob),
+     prc_auc         = estimates_rf_tbl %>% pr_auc(truth, class_prob),
+     precision       = estimates_rf_tbl %>% precision(truth, estimate),
+     recall          = estimates_rf_tbl %>% recall(truth, estimate)
+ )

# A tibble: 1 x 4
    auc prc_auc precision recall
  <dbl>   <dbl>     <dbl>  <dbl>
1 0.827   0.216     0.232  0.614

02 - February

## CONFUSION TABLE
> estimates_rf_tbl %>% conf_mat(truth, estimate)
          Truth
Prediction     No    Yes
       No  247106   6477
       Yes  34636  10655
> 

 ## OVER MODEL PERFORMANCE
> tibble(
+     auc             = estimates_rf_tbl %>% roc_auc(truth, class_prob),
+     prc_auc         = estimates_rf_tbl %>% pr_auc(truth, class_prob),
+     precision       = estimates_rf_tbl %>% precision(truth, estimate),
+     recall          = estimates_rf_tbl %>% recall(truth, estimate)
+ )
# A tibble: 1 x 4
    auc prc_auc precision recall
  <dbl>   <dbl>     <dbl>  <dbl>
1 0.839   0.218     0.235  0.622

03 - March

## CONFUSION TABLE
> estimates_rf_tbl %>% conf_mat(truth, estimate)
          Truth
Prediction     No    Yes
       No  250869   4662
       Yes  45114   8709

 ## OVER MODEL PERFORMANCE
> tibble(
+     auc             = estimates_rf_tbl %>% roc_auc(truth, class_prob),
+     prc_auc         = estimates_rf_tbl %>% pr_auc(truth, class_prob),
+     precision       = estimates_rf_tbl %>% precision(truth, estimate),
+     recall          = estimates_rf_tbl %>% recall(truth, estimate)
+ )
# A tibble: 1 x 4
    auc prc_auc precision recall
  <dbl>   <dbl>     <dbl>  <dbl>
1 0.829   0.151     0.162  0.651

I am sort of indecisive in my choice. Hope that someone can guide me in an engaging and insightful way, and I look forward to your feedback!

Thanks in advance

Answer 1

I developed a churn scoring for a company months ago.

Personnaly what i did was: Train on January, evaluation on Feb/May/Sep and for each compute the lift, the gain and the % of each deciles.

All these metrics must stay stable through time, lift and gain for the performance, % for the stability.

Theses metrics can be monitored each month on a dashboard, in comparison of the training metrics. Easily understand by anybody.

They were really satisfied to have a modeling score easily readable, I think metrics like ROC, PCC and co should stay in your process modeling but never to be communicate unless you want to lose your audience.

After that, when you did this kind of modeling, you have to think about an incremental strategy,

You should look at this paper that gives you multipled methods to do so: https://support.sas.com/resources/papers/proceedings13/096-2013.pdf