I applied different classification algorithms in combination with different sampling techniques to a dataset and I get > 100 different models with different performances.

I can choose a model for high precision or for high recall, but obviously not both at the same time.

Is there an approach/method/function out there where I can penalize either false positives or false negatives more – based on what is more/less important to me – so I can choose the perfect model out of all the ones I calculated?

  • 1
    $\begingroup$ Usually, you as a user should be able to define a suitable utility function for your application. Standard metrics, like $F_\beta$ mentioned below, will rarely match what matters for your task. $\endgroup$ – Marc Claesen Nov 25 '15 at 9:52

Sure. You can use Fbeta score.

enter image description here

Beta = 1 means you value precision and recall equally, higher beta (beta > 1) means you value precision more then recall.

More on wiki: F1 score

| cite | improve this answer | |
  • $\begingroup$ Is there not an issue with overfitting here, too, so that picking the best model on this basis does not necessarily result in good out of sample prediction? Would potentially AIC model averaging not do better? $\endgroup$ – Björn Nov 25 '15 at 10:16
  • $\begingroup$ Hopefully "performances" in the OP refers to out-of-sample performances. $\endgroup$ – rolando2 Nov 25 '15 at 12:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.