How to control trade-off between precision and recall?

I applied different classification algorithms in combination with different sampling techniques to a dataset and I get > 100 different models with different performances.

I can choose a model for high precision or for high recall, but obviously not both at the same time.

Is there an approach/method/function out there where I can penalize either false positives or false negatives more – based on what is more/less important to me – so I can choose the perfect model out of all the ones I calculated?

• Usually, you as a user should be able to define a suitable utility function for your application. Standard metrics, like $F_\beta$ mentioned below, will rarely match what matters for your task. – Marc Claesen Nov 25 '15 at 9:52