Let's say I have a data set where half of the data points are labelled as positive and half are labelled as negative. My task is to create a classifier which recognises when a sample from the dataset is positive.
The most useless classifier I could come up with would be to flip a fair coin whenever I get a sample to decide if the sample is positive or negative. One way to quantify the performance of that classifier is the f1 score which is in expectation 0.5 for a large dataset since recall is expected to be 0.5 and precision is expected to be 0.5. Additionally the f1 score should in principle be concentrated. That is intuitively the baseline f1 score I would compare everything to.
Now instead of that useless classifier I could create another classifier which in my opinion is just as useless where I simply say that all the samples I receive are positive. In that case my precision is still 0.5 but the recall is 1. This leads to an f1 score of 2/3 so if I use the f1 score as a measure to decide which classifier is better I should pick this one instead of the random one.
This is perhaps more of a philosophical pondering but in my opinion a way to select a classifier should not distinguish between those two classifiers (why should I prefer the second option?). Therefore I wanted to ask about alternatives that do pass such a test. I'd prefer a single number which is a function of true positive rate, false positive rate, true negative rate and false negative rate.