0
$\begingroup$

I'm working on a credit binary classification task.For this business is something usual to meassure model's performance from two metrics: ROC AUC and KS .This sounds reasonable until I have to choose between two models with "contradictory" results. Let's say I have the following:

Model 1 : AUC: .90 & KS: .70 Model 2 : AUC: .85 & KS: .80

I have been thinking on a way to combine these two (or even more metrics) to make a final decision on which model is better.

Question:

1. What would be a correct way of combining multiple metrics to make a decision of a model to be selected?

2. What are the characteristics this combination should fulfill? (limit to infinity 1 , limit to minus infinity 0)

EDIT:

I'm looking for either a formula to combine multiple metrics into one score to be able to make decisions based on that or a methodology to take into consideration more than one metric to make decisions

$\endgroup$
3
  • $\begingroup$ Are you looking for some sort of formula that you can apply, or more general ideas for an approach to consider the AUC and KS scores in model evaluation? $\endgroup$
    – hamedbh
    Jul 18 '19 at 14:57
  • $\begingroup$ @hamedbh It would be great having both approaches $\endgroup$
    – user251839
    Jul 18 '19 at 15:27
  • $\begingroup$ How are you using KS? $\endgroup$
    – Dave
    May 11 '20 at 2:16
0
$\begingroup$

You could take a more general approach overall. Depending on the computational burden of the model you could use Leave One Out Cross Validation. This approach is desirable when the models are not nested, or when metrics provide contradictory results, as is your case.

$\endgroup$
1
  • $\begingroup$ How would you evaluate performance on the left-out data? $\endgroup$
    – Dave
    May 11 '20 at 2:17
0
$\begingroup$

Are the two models parametric? If yes, you can weigh them with respective likelihood scores. This is equivalent to Bayesian aggregation (weighing with Bayesian scores) when the prior distribution over candidate models is uniform. Bayesian aggregation is a popular, scientific way of combining models when there is belief that each of them carries some useful information (when one model does not stochastically dominate another).

For example, my suggestion works if you have performed classification using logit or probit models. On the other hand, if you are using tree-based methods an extension of the aforementioned ideas is necessary.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy