4
$\begingroup$

I know how to calculate the baseline for the accuracy of a binary classification problem: I simply always predict the majority class, e.g. if there is 94% True values and 6% False values, my baseline accuracy is 94%.

However, in such an unbalanced dataset, F1 score is a much better metric to measure performance of a classifier. However, how do I get the baseline for the F1 score?

My intuition is to either

a) Always predict "True" b) Randomly predict "True" with a certain likelyhood c) Like b), but the likelihood being the rate of True values in the data.

The problem with b) and c) is that the F1 score will vary wildly in very unbalanced datasets, since a single hit can make all the difference. I would therefore go with a), is that correct?

$\endgroup$

1 Answer 1

10
$\begingroup$

Technically, you can choose whatever fits your needs as a baseline. Its simply a lower bound on performance for model evaluation. When the baseline is defined as a dummy predictor, a learned model is of course expected to outperform it, otherwise you know something is wrong with the learning pipeline.

Let's assume that we have marginal probabilities $p(y=1) = r$ and $p(y=0) = 1-r$. We use TP, TF, FP and FN to indicate the true/false positive/negative cases.

You proposed 3 dummy predictors and here are the F1-scores:

  • Always true: The precision simplifies to $r$ and the recall is obviously 1. Thus, F1-score is $\frac{ 2r}{r + 1}$

  • Predict 1 with some probability $q$: The precision simplifies to $r$ (subject to $q > 0$) and the recall simplifies to $q$. In this case, the F1-score is $\frac{ 2rq}{r + q}$, which is maximized when $q = 1$ (always predicting true)

  • Predict 1 with probability $q=r$: In this case, the F1-score becomes $r$

Basically, this means that the best dummy classifier (among the 3) with respect to the F1-score is to always predict true. Using it as your baseline means that your F1-score must be above $\frac{ 2r}{r + 1}$.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.