How to combine accuracy scores of two models. Same samples with different output values

Question

I have two classification models that each classify the same input data to two distinct sets of output values. The outputs of the two models will be used to create a taxonomy, which functions as the overall output value.

Equivalent example:

Inputs are different kinds of fruit and vegetables.

Output for model A is { fruit, vegetable } Output for model B is { small, large }

So the overall model would work like this:

• input: cabbage -> output: large vegetable
• input: berry -> output: small fruit

How can I combine the two accuracies of the models to achieve an overall accuracy.

Intuitively I would just multiply the two accuracies. Is this intuition correct or is there a better way?

Thanks in advance.

Answer 1

You cannot just multiply the two accuracies. Imagine both models to be correct on exactly the same inputs, which is maybe 50% of all inputs, then the two models would each have an accuracy of 0.5, as would their combination, while the product results in an accuracy of 0.25.

The total accuracy depends on the overlap of the set $S_1$ of inputs that the first model gets right and the set $S_2$ of inputs the second model gets right. Using the symbol $acc_{comb}$ for the accuracy of the combined model, the symbol $\#S$ for the size of a set $S$, and letting $M$ be the set of all inputs, then: $$ acc_{comb} = \frac{\#(S_1\cap S_2)}{\#M}. $$ You cannot compute $acc_{comb}$ from the accuracies of the two models, because the same accuracies of the two models can lead to different accuracies of the combined model given different overlaps.