1
$\begingroup$

I'm working on a credit binary classification task.For this business is something usual to meassure model's performance from two metrics: ROC AUC and KS .This sounds reasonable until I have to choose between two models with "contradictory" results. Let's say I have the following:

Model 1 : AUC: .90 & KS: .70 Model 2 : AUC: .85 & KS: .80

I have been thinking on a way to combine these two (or even more metrics) to make a final decision on which model is better.

Question:

1. What would be a correct way of combining multiple metrics to make a decision of a model to be selected?

2. What are the characteristics this combination should fulfill? (limit to infinity 1 , limit to minus infinity 0)

EDIT:

I'm looking for either a formula to combine multiple metrics into one score to be able to make decisions based on that or a methodology to take into consideration more than one metric to make decisions

$\endgroup$
4
  • $\begingroup$ Are you looking for some sort of formula that you can apply, or more general ideas for an approach to consider the AUC and KS scores in model evaluation? $\endgroup$
    – hamedbh
    Commented Jul 18, 2019 at 14:57
  • $\begingroup$ @hamedbh It would be great having both approaches $\endgroup$
    – user251839
    Commented Jul 18, 2019 at 15:27
  • $\begingroup$ How are you using KS? $\endgroup$
    – Dave
    Commented May 11, 2020 at 2:16
  • $\begingroup$ Take a look at the multi-metric evaluation on cross_val_score and GridSearchCV here. I believe this is exactly what you are looking for. $\endgroup$
    – Ramm
    Commented Oct 17, 2022 at 13:56

2 Answers 2

0
$\begingroup$

You could take a more general approach overall. Depending on the computational burden of the model you could use Leave One Out Cross Validation. This approach is desirable when the models are not nested, or when metrics provide contradictory results, as is your case.

$\endgroup$
1
  • $\begingroup$ How would you evaluate performance on the left-out data? $\endgroup$
    – Dave
    Commented May 11, 2020 at 2:17
0
$\begingroup$

Are the two models parametric? If yes, you can weigh them with respective likelihood scores. This is equivalent to Bayesian aggregation (weighing with Bayesian scores) when the prior distribution over candidate models is uniform. Bayesian aggregation is a popular, scientific way of combining models when there is belief that each of them carries some useful information (when one model does not stochastically dominate another).

For example, my suggestion works if you have performed classification using logit or probit models. On the other hand, if you are using tree-based methods an extension of the aforementioned ideas is necessary.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.