1
$\begingroup$

I have two classification models that each classify the same input data to two distinct sets of output values. The outputs of the two models will be used to create a taxonomy, which functions as the overall output value.

Equivalent example:

Inputs are different kinds of fruit and vegetables.

Output for model A is { fruit, vegetable } Output for model B is { small, large }

So the overall model would work like this:

  • input: cabbage -> output: large vegetable
  • input: berry -> output: small fruit

How can I combine the two accuracies of the models to achieve an overall accuracy.

Intuitively I would just multiply the two accuracies. Is this intuition correct or is there a better way?

Thanks in advance.

$\endgroup$
4
  • $\begingroup$ This depends very much on how exactly you combine those two models. $\endgroup$
    – frank
    Jun 17 at 8:49
  • $\begingroup$ What do you mean? $\endgroup$ Jun 17 at 8:55
  • $\begingroup$ You say "The outputs of the two models will be used to create a taxonomy, which functions as the overall output value." I refer to this combining of the two models. $\endgroup$
    – frank
    Jun 17 at 8:58
  • $\begingroup$ I have updated my question with an equivalent example, hope this answers your question. $\endgroup$ Jun 17 at 9:01

1 Answer 1

1
$\begingroup$

You cannot just multiply the two accuracies. Imagine both models to be correct on exactly the same inputs, which is maybe 50% of all inputs, then the two models would each have an accuracy of 0.5, as would their combination, while the product results in an accuracy of 0.25.

The total accuracy depends on the overlap of the set $S_1$ of inputs that the first model gets right and the set $S_2$ of inputs the second model gets right. Using the symbol $acc_{comb}$ for the accuracy of the combined model, the symbol $\#S$ for the size of a set $S$, and letting $M$ be the set of all inputs, then: $$ acc_{comb} = \frac{\#(S_1\cap S_2)}{\#M}. $$ You cannot compute $acc_{comb}$ from the accuracies of the two models, because the same accuracies of the two models can lead to different accuracies of the combined model given different overlaps.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.