1
$\begingroup$

Let's say, I have a binary classifier. Usually, if a model doesn't learn anything useful, the accuracy would be around 50%. Anything above 50 is better than a random guess.

My question is: what are some situations where the model will show below 50% accuracy (or some performance metric), let's say 30%? Doesn't that mean, I can just always reverse the answer and get 70% accuracy?

Can this happen (a model showing some 20-30ish % performance)? If yes, what has the model learnt actually?

Improve this question
$\endgroup$
2
  • $\begingroup$ "Anything above 50 is better than a random guess." that is only true if you assume that both classes occur equally frequent in the dataset $\endgroup$
    – Janosch
    Commented Apr 5, 2023 at 10:54
  • $\begingroup$ yes, I assume the datasets are balanced. Thanks for pointing it out. $\endgroup$
    – Zabir Al Nazi Nabil
    Commented Apr 5, 2023 at 14:44

1 Answer 1

Reset to default
1
$\begingroup$

For a binary classification problem, you could get accuracy equal to the proportion of the most frequent class by always predicting this class. If the data is perfectly balanced if you either always predict "0" or "1" the accuracy would be 50%. Anything lower than this means that you would be better just by predicting always one class. If 90% of your data is of one class, if you predict this class for all the data you get 90% accuracy, so you should not aim lower.

Reversing the predicted labels could technically solve the problem, but doing it without diagnosing why the performance was so poor would not be a great idea. You should also see how it affects other metrics (and in general look at other metrics).

See also Why is accuracy not the best measure for assessing classification models?, as accuracy itself is a rather poor metric that can easily mislead you about the quality of the model.

Improve this answer
$\endgroup$
2
  • $\begingroup$ +1. I understand the frequency of labels can definitely impact the performance score (if the train distribution is 80:20 and in test it's 20:80, the model will show a very different performance in train and test), and yes accuracy is never a good metric but I only mention accuraccy since it's the simplest one to explain and if you just reverse the labels you get (1-previous accuracy). I am mostly interested in the case: can a model learn a mapping where it produces an accuracy of 25% or 30% in a perfectly balanced dataset. Since in that case: we can just reverse the label and get 75% - 70% acc $\endgroup$
    – Zabir Al Nazi Nabil
    Commented Apr 5, 2023 at 16:51
  • $\begingroup$ @ZabirAlNazi yes. Accuracy is mean(actual == predicted). If you inverse the labels it's mean(actual == !predicted) == mean(actual != predicted) (assuming you are using language like R that would dynamically cast integers to booleans). $\endgroup$
    – Tim
    Commented Apr 5, 2023 at 18:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.