I applied different classification algorithms in combination with different sampling techniques to a dataset and I get > 100 different models with different performances.

I can choose a model for high precision or for high recall, but obviously not both at the same time.

Is there an approach/method/function out there where I can penalize either false positives or false negatives more – based on what is more/less important to me – so I can choose the perfect model out of all the ones I calculated?

  • 1
    $\begingroup$ Usually, you as a user should be able to define a suitable utility function for your application. Standard metrics, like $F_\beta$ mentioned below, will rarely match what matters for your task. $\endgroup$ Nov 25, 2015 at 9:52

1 Answer 1


Sure. You can use Fbeta score.

enter image description here

Beta = 1 means you value precision and recall equally, higher beta (beta > 1) means you value precision more then recall.

More on wiki: F1 score

  • $\begingroup$ Is there not an issue with overfitting here, too, so that picking the best model on this basis does not necessarily result in good out of sample prediction? Would potentially AIC model averaging not do better? $\endgroup$
    – Björn
    Nov 25, 2015 at 10:16
  • $\begingroup$ Hopefully "performances" in the OP refers to out-of-sample performances. $\endgroup$
    – rolando2
    Nov 25, 2015 at 12:26
  • $\begingroup$ @Yurii , wouldn't higher $\beta$ mean you value value recall more than precision? If you set $\beta$ to be a very high value, the $\beta$ and precision values will get cancelled out. $\endgroup$
    – abhishek
    Oct 1 at 18:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.