6
$\begingroup$

Lets say I have two multi-class models (A and B) that are predicting whether or not a set of inputs belong to one of 5 classes. Each models predictions are probabilities that sum to 1. As an example, imagine the following:

+---------+---------+---------+--------+
|         | Model A | Model B | Result |
+---------+---------+---------+--------+
| Class 1 | 0.2     | 0.4     | ?      |
+---------+---------+---------+--------+
| Class 2 | 0.3     | 0.3     | ?      |
+---------+---------+---------+--------+
| Class 3 | 0.15    | 0.2     | ?      |
+---------+---------+---------+--------+
| Class 4 | 0.25    | 0.05    | ?      |
+---------+---------+---------+--------+
| Class 5 | 0.1     | 0.05    | ?      |
+---------+---------+---------+--------+

How would I go about combining these probabilities into one single probability that still sums to one?

$\endgroup$

1 Answer 1

8
$\begingroup$

The simplest approach would be to just take an average of the predictions for each class. You can use a weight.

Suppose the first model gives predictions $p_1, \dots, p_5$ and the second gives $q_1, \dots, q_5$. Then

$$\sum_{i=1}^5 p_i=\sum_{i=1}^5 q_i = 1.$$

Take any weight $0<w<1$, define the combined prediction by $r_i := wp_i+(1-w)q_i$. Then

$$ \sum_{i=1}^5 r_i = \sum_{i=1}^5\big(wp_i+(1-w)q_i\big) = w\sum_{i=1}^5 p_i+(1-w)\sum_{i=1}^5 q_i = w+(1-w) = 1. $$

So your predictions again sum up to 1. This also works for more than two classifiers.

As weights, you could use past performance of your two classifiers. Or take the unweighted average, $w=\frac{1}{2}$, which is frequently better than trying to estimate "optimal" weights (Claeskens et al., 2016, IJF).

Alternatively, you can use any other method to combine your predictions, and just renormalize the results afterwards to sum to 1.

$\endgroup$
1
  • $\begingroup$ Amazing and concise answer, thank you very much! $\endgroup$
    – Will.Evo
    Commented Aug 18, 2020 at 15:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.