Combining non-probabilistic models for higher quality predictions

Question

Three people have independently developed models for predicting a coin flip. They take into account the launch angle, launch force, rate of spin, and various other factors to produce predictive models of varying quality. After a million flips:

1. Model A predicts the result correctly 60% of the time
2. Model B predicts the result correctly 70% of the time
3. Model C predicts the result correctly 80% of the time

It seems to me that while C is clearly the best model, models A and B are not also worthless. I suspect that, if all three models point to the same result then that result is in actuality much more likely than any one individual model would suggest. Likewise I suspect that if Models A and B predict Heads while Model C predicts Tails, the actual probability of Tails is less than the historical 80% would suggest.

I ran a quick and dirty monte carlo simulation and after 5mil rounds found that, with surprising consistently, when Model A and Model B predicted Heads while Model C predicted Tails, Model C was correct only about 54% of the time.

import random

def Predicts(Model, Value):
    if Value > Model:
        return False    
    else:
        return True
def simulation():
    numberOfEvents = 0
    numberOfTimesABCorrectlyCounteredC = 0
    A = 60
    B = 70
    C = 80
    while numberOfEvents < 10000:
        AValue = random.randint(1,100)
        BValue = random.randint(1,100)
        CValue = random.randint(1,100)
        APred = Predicts(A, AValue)
        BPred = Predicts(B, BValue)
        CPred = Predicts(C, CValue)
        if APred == BPred and APred !=CPred:
            numberOfEvents = numberOfEvents + 1
            if CPred==False:
                numberOfTimesABCorrectlyCounteredC = numberOfTimesABCorrectlyCounteredC + 1
    return numberOfTimesABCorrectlyCounteredC


count = 0
number = 0
while count < 100:
    number = number + simulation()
    count = count + 1

print("After 1,000,000 events AB correctly countered C this many times:")
print(number)

Given models that predict a binary outcome (and not a probability), and the historical accuracy of those models based on a large pool of examples, is there a way to combine the models such that their overall predictive power is greater than any one individual model?

Side note: My stat-fu is not the strongest and I wouldn't be surprised if I get torched for something I missed (maybe in my tags of assumptions). No hard feelings, and either way, I appreciate your feedback.

Answer 1

There are many ways to combine the 3 models. If you have the response variable and predictions for a good number of observations, you can combine them in a way that better predicts the response variable.

One can average them, build a weighted average from a logistic regression, or a tree based approach. Those are called ensembles and are quite common. I know of some evidence that those simple techniques can work quite well, but the possibilities are endless and can get quite messy when it comes to ensembles.

Answer 2

Let's assume that the coin is fair, $p_{heads} = p_{tails} = 0.5$. Let's assume that the models are independent in their predictions (*).

Bayesian approach

Then we can make our decisions based on a Bayesian posterior prediction.

The posterior odds ratio for the true coin being heads (H) and the true coin being tails (T) are the likelihood ratio multiplied with the prior odds ratio

$$\frac{P(H|A,B,C)}{P(T|A,B,C)} = \frac{P(A,B,C[H)}{P(A,B,C|T)} \frac{P(H)}{P(T)} = \frac{P(A[H)}{P(A|T)} \frac{P(B[H)}{P(B|T)} \frac{P(C|H)}{P(C|T)}$$

The likelihood ratios for observations A, B, C (observation is whether the model says heads or whether the model says tails) are

$$\frac{P(A|H)}{P(A|T)} = \begin{cases} 0.6/0.4 &\quad \text{if $A=$heads}\\ 0.4/0.6 &\quad \text{if $A=$tails} \end{cases}$$

$$\frac{P(B|H)}{P(B|T)} = \begin{cases} 0.7/0.3 &\quad \text{if $B=$heads}\\ 0.3/0.7 &\quad \text{if $B=$tails} \end{cases}$$

$$\frac{P(C|H)}{P(C|T)} = \begin{cases} 0.8/0.2 &\quad \text{if $C=$heads}\\ 0.2/0.8 &\quad \text{if $C=$tails} \end{cases}$$

• If you have A and B tails and C heads then the posterior odds for heads will be $$\frac{0.8\cdot 0.3\cdot 0.4}{0.2\cdot 0.7\cdot 0.3} = 8/7 \approx 1.14$$

  And, conditional on this observation, you will be correct only $1.14/(1+1.14)\approx0.533$ of the time. Which is close to your simulation. (More precisely it is 8/15)

• When you observe C heads then the posterior odds for heads:tails will be >1 no matter what the predictions are from A and B. The story will be different when you have different models (different in number and/or different in success rates).

Frequentist approach

From a frequentist perspective we will partition the model outcomes in two groups such that our probability for being correct is maximized.

Observation ABC   P if heads    P if tails
HHH               0.366         0.024
THH               0.224         0.036
HTH               0.114         0.056
TTH               0.096         0.084

HHT               0.084         0.096
THT               0.056         0.114
HTT               0.036         0.224
TTT               0.024         0.366

In the table above we ordered the outcomes based on probability. For example, if the true coin is heads then the probability that the models A,B and C say all heads (HHH) is 0.366. We draw a line in the middle to make our decision boundary (such that the probability of error/success is equal for when the true value is heads as when the true value is tails). Note that, conditional on heads being correct, the case with single heads TTH is more likely than this case with two heads HHT. And that is why TTH makes us choose heads and HHT not.

With the table above we can compute the success rate of majority voting that Stephan Kolassa mentions in the comments. It'll be 0.366+0.224+0.114+0.084 = 0.788

(*)which is not realistic, but it makes the math a lot easier, and also without specifying correlation or joint probability for the models, it is very difficult to think of a computation that is applicable.