0
$\begingroup$

I have a customer churn model, which classifies people who are going to leave (Yes) from those who are staying (No). I trained my model using 10-fold Cross-validation and I used AUC under ROC curver as a metric for my model performance, to begin with.

However, I went through many nice readings on the subject and I got difference at times confusing suggestions. Which can be summarised as follow.

  • Accuracy will not always be the metric.
  • Precision and recall are often in tension. That is, improving precision typically reduces recall and vice versa.
  • AUC-ROC curve is one of the most commonly used metrics to evaluate the performance of machine learning algorithms.
  • ROC Curves summarise the trade-off between the true positive rate and false positive rate for a predictive model using different probability thresholds.
  • The ROC curve can be used to choose the best operating point.

$Question:$ Which metric I should I give to business telling them my model is doing a good job

$Question:$ I need to give some number of business KPI; which should be interpretable to them. What could be my choice to business?

DEPLOYED MODEL PERFORMANCE

01 - January

## CONFUSION TABLE
> estimates_rf_tbl %>% conf_mat(truth, estimate)
          Truth
Prediction     No    Yes
       No  229493   6604
       Yes  34687  10497

## OVER MODEL PERFORMANCE
> tibble(
+     auc             = estimates_rf_tbl %>% roc_auc(truth, class_prob),
+     prc_auc         = estimates_rf_tbl %>% pr_auc(truth, class_prob),
+     precision       = estimates_rf_tbl %>% precision(truth, estimate),
+     recall          = estimates_rf_tbl %>% recall(truth, estimate)
+ )

# A tibble: 1 x 4
    auc prc_auc precision recall
  <dbl>   <dbl>     <dbl>  <dbl>
1 0.827   0.216     0.232  0.614

02 - February

## CONFUSION TABLE
> estimates_rf_tbl %>% conf_mat(truth, estimate)
          Truth
Prediction     No    Yes
       No  247106   6477
       Yes  34636  10655
> 

 ## OVER MODEL PERFORMANCE
> tibble(
+     auc             = estimates_rf_tbl %>% roc_auc(truth, class_prob),
+     prc_auc         = estimates_rf_tbl %>% pr_auc(truth, class_prob),
+     precision       = estimates_rf_tbl %>% precision(truth, estimate),
+     recall          = estimates_rf_tbl %>% recall(truth, estimate)
+ )
# A tibble: 1 x 4
    auc prc_auc precision recall
  <dbl>   <dbl>     <dbl>  <dbl>
1 0.839   0.218     0.235  0.622

03 - March

## CONFUSION TABLE
> estimates_rf_tbl %>% conf_mat(truth, estimate)
          Truth
Prediction     No    Yes
       No  250869   4662
       Yes  45114   8709

 ## OVER MODEL PERFORMANCE
> tibble(
+     auc             = estimates_rf_tbl %>% roc_auc(truth, class_prob),
+     prc_auc         = estimates_rf_tbl %>% pr_auc(truth, class_prob),
+     precision       = estimates_rf_tbl %>% precision(truth, estimate),
+     recall          = estimates_rf_tbl %>% recall(truth, estimate)
+ )
# A tibble: 1 x 4
    auc prc_auc precision recall
  <dbl>   <dbl>     <dbl>  <dbl>
1 0.829   0.151     0.162  0.651

I am sort of indecisive in my choice. Hope that someone can guide me in an engaging and insightful way, and I look forward to your feedback!

Thanks in advance

$\endgroup$
0
$\begingroup$

I developed a churn scoring for a company months ago.

Personnaly what i did was: Train on January, evaluation on Feb/May/Sep and for each compute the lift, the gain and the % of each deciles.

All these metrics must stay stable through time, lift and gain for the performance, % for the stability.

Theses metrics can be monitored each month on a dashboard, in comparison of the training metrics. Easily understand by anybody.

They were really satisfied to have a modeling score easily readable, I think metrics like ROC, PCC and co should stay in your process modeling but never to be communicate unless you want to lose your audience.

After that, when you did this kind of modeling, you have to think about an incremental strategy,

You should look at this paper that gives you multipled methods to do so: https://support.sas.com/resources/papers/proceedings13/096-2013.pdf

$\endgroup$
  • $\begingroup$ thanks very much for your detailed information. that's interesting insight but partly answer what I need. but I also have to tell them how good is model pin predicting churner vs non-churner. what metrics I can communicate with management about my model alone? $\endgroup$ – James Taylor Jul 1 at 8:12
  • $\begingroup$ lift and cumulative gains is what you need to communicate. the lift (a lift above 1.2 is good, above 2 is great, above 3 is super really good) will tell you how much you target better than having non modeling and cumulative gains will tell you how much churner you target in your highest deciles. that's what it's important in business, not the performance of good classification between churner vs non churner that has no business sense. $\endgroup$ – josef_joestarr Jul 1 at 23:27
  • $\begingroup$ the thing that will really tell how good is your model is those metrics above + an incremental response analysis of how many churners you saved with your modeling with targeting (and you can add a value for those churner if you have in your data how much they bring to the company) $\endgroup$ – josef_joestarr Jul 1 at 23:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.