The title says it all: Is F-score the same as accuracy when there are only two classes of equal sizes?
For my specific case, I have measurements of a group of people under two different situations and wish to be able to classify based on the measurements. I therefore have 50 measurements for situation A, and 50 for situation B. In this situation, will F-score always be the same as accuracy, and, actually, precision and recall too?
(And if so or not, please explain why.)
No.
The F-score is a weighted average of precision (accuracy among predicted positives) and recall (accuracy among actual positives). Neither measure says anything at all about the group of true negatives and how well you classified them, but overall accuracy evaluates performance across all samples. Any metric that totally ignores the true negative group is ignoring data required to evaluate accuracy.
Suppose you have 100 items, 50 of which are actually positive and 50 of which are actually negative. You predict 10 items as positive and get none of them right, and 90 items as negative and get 40 of them right. Your precision is 0 (none of your positive predictions were right) , your recall is 0 (you haven't identified any actual positives), so any F-score must also be 0, despite the fact that your accuracy is 40%.
NO, see the counterexample below
Classify every instance the same way. You have a recall of $1$, a precision of $0.5$, and an accuracy of $0.5$. However, your $F_1$ score is:
$$ F_1 = 2\dfrac{\text{precision}\times\text{recall}}{\text{precision}+\text{recall}} = 2\dfrac{0.5\times 1}{0.5 + 1} = \dfrac{1}{1.5} = \dfrac{2}{3}\ne 0.5=\text{Accuracy} $$
A simulation can show many counterexamples where accuracy, recall, precision, and $F_1$ are different, despite the balance.
library(MLmetrics)
set.seed(2024)
R <- 1000
N0 <- N1 <- 50
N <- N0 + N1
y_true <- rep(c(0, 1), c(N0, N1))
accs <- precs <- recalls <- f1s <- rep(NA, R)
for (i in 1:R){
y_pred <- rbinom(N, 1, 0.5)
accs[i] <- MLmetrics::Accuracy(y_pred, y_true)
precs[i] <- MLmetrics::Precision(y_pred, y_true)
recalls[i] <- MLmetrics::Recall(y_pred, y_true)
f1s[i] <- MLmetrics::F1_Score(y_true, y_pred, 1)
}
d <- data.frame(
Accuracy = accs,
Precision = precs,
Recall = recalls,
F1 = f1s
)
plot(d)
Equality would require the points to lie along the line $y = x,$ which is false for all of the pairs shown above.
