3
$\begingroup$

I applied different classification algorithms in combination with different sampling techniques to a dataset and I get > 100 different models with different performances.

I can choose a model for high precision or for high recall, but obviously not both at the same time.

Is there an approach/method/function out there where I can penalize either false positives or false negatives more – based on what is more/less important to me – so I can choose the perfect model out of all the ones I calculated?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Usually, you as a user should be able to define a suitable utility function for your application. Standard metrics, like $F_\beta$ mentioned below, will rarely match what matters for your task. $\endgroup$
    – Marc Claesen
    Commented Nov 25, 2015 at 9:52

1 Answer 1

Reset to default
1
$\begingroup$

Sure. You can use Fbeta score.

enter image description here

Beta = 1 means you value precision and recall equally, higher beta (beta > 1) means you value precision more then recall.

More on wiki: F1 score

Improve this answer
$\endgroup$
3
  • $\begingroup$ Is there not an issue with overfitting here, too, so that picking the best model on this basis does not necessarily result in good out of sample prediction? Would potentially AIC model averaging not do better? $\endgroup$
    – Björn
    Commented Nov 25, 2015 at 10:16
  • $\begingroup$ Hopefully "performances" in the OP refers to out-of-sample performances. $\endgroup$
    – rolando2
    Commented Nov 25, 2015 at 12:26
  • $\begingroup$ @Yurii , wouldn't higher $\beta$ mean you value value recall more than precision? If you set $\beta$ to be a very high value, the $\beta$ and precision values will get cancelled out. $\endgroup$
    – abhishek
    Commented Oct 1, 2023 at 18:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.