Skip to main content
3 of 3
deleted 4 characters in body

Combining non-probabilistic models for higher quality predictions

Three people have independently developed models for predicting a coin flip. They take into account the launch angle, launch force, rate of spin, and various other factors to produce predictive models of varying quality. After a million flips:

  1. Model A predicts the result correctly 60% of the time
  2. Model B predicts the result correctly 70% of the time
  3. Model C predicts the result correctly 80% of the time

It seems to me that while C is clearly the best model, models A and B are not also worthless. I suspect that, if all three models point to the same result then that result is in actuality much more likely than any one individual model would suggest. Likewise I suspect that if Models A and B predict Heads while Model C predicts Tails, the actual probability of Tails is less than the historical 80% would suggest.

I ran a quick and dirty monte carlo simulation and after 5mil rounds found that, with surprising consistently, when Model A and Model B predicted Heads while Model C predicted Tails, Model C was correct only about 54% of the time.

import random

def Predicts(Model, Value):
    if Value > Model:
        return False    
    else:
        return True
def simulation():
    numberOfEvents = 0
    numberOfTimesABCorrectlyCounteredC = 0
    A = 60
    B = 70
    C = 80
    while numberOfEvents < 10000:
        AValue = random.randint(1,100)
        BValue = random.randint(1,100)
        CValue = random.randint(1,100)
        APred = Predicts(A, AValue)
        BPred = Predicts(B, BValue)
        CPred = Predicts(C, CValue)
        if APred == BPred and APred !=CPred:
            numberOfEvents = numberOfEvents + 1
            if CPred==False:
                numberOfTimesABCorrectlyCounteredC = numberOfTimesABCorrectlyCounteredC + 1
    return numberOfTimesABCorrectlyCounteredC


count = 0
number = 0
while count < 100:
    number = number + simulation()
    count = count + 1

print("After 1,000,000 events AB correctly countered C this many times:")
print(number)

Given models that predict a binary outcome (and not a probability), and the historical accuracy of those models based on a large pool of examples, is there a way to combine the models such that their overall predictive power is greater than any one individual model?

Side note: My stat-fu is not the strongest and I wouldn't be surprised if I get torched for something I missed (maybe in my tags of assumptions). No hard feelings, and either way, I appreciate your feedback.