1
$\begingroup$

If I have 3 variables: Jimmy, Tommy and Jenny, and I trained 3 "one VS rest" models and got the following:

  • model accuracy for Jimmy = 99%
  • model accuracy for Tommy = 95%
  • model accuracy for Jenny = 70%

Will my combined model accuracy be $0.99 \cdot 0.95 \cdot 0.70$?

$\endgroup$
3
$\begingroup$

Almost certainly not. Take any example, like 100 of each group. You correctly classified 99 instances of the first group, 95 of the second and 70 of the third group. Your overall accuracy is $$\frac{99+95+70}{100+100+100}=0.88\ \neq 0.66 = 0.99\times 0.95\times 0.70. $$ However, we do have $$ 0.88=\frac{0.99+0.95+0.70}{3}, $$ so a different hypothesis would be that overall accuracy is the average of the group accuracies.

And this does not necessarily hold, either. Suppose you have 100 Jimmies (99 correctly classified), 20 Tommies (19 correctly classified, and 10 Jennies (7 correctly classified). Your overall accuracy is $$\frac{99+19+7}{100+20+10}=0.96. $$

The correct combination is the weighted average, where each group's accuracy is weighted by the total group size.

(Also, accuracy is not very useful in the first place.)

$\endgroup$
1
  • $\begingroup$ easy understand explanation! thanks! $\endgroup$ – GarlicSTAT Jul 29 '20 at 21:11
0
$\begingroup$

The difference is almost into micro and macro averaging. You can look at this

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.