Three people have independently developed models for predicting a coin flip. They take into account the launch angle, launch force, rate of spin, and various other factors to produce predictive models of varying quality. After a million flips:
- Model A predicts the result correctly 60% of the time
- Model B predicts the result correctly 70% of the time
- Model C predicts the result correctly 80% of the time
It seems to me that while C is clearly the best model, models A and B are not also worthless. I suspect that, if all three models point to the same result then that result is in actuality much more likely than any one individual model would suggest. Likewise I suspect that if Models A and B predict Heads while Model C predicts Tails, the actual probability of Tails is less than the historical 80% would suggest.
I ran a quick and dirty monte carlo simulation and after 5mil rounds found that, with surprising consistently, when Model A and Model B predicted Heads while Model C predicted Tails, Model C was correct only about 54% of the time.
import random
def Predicts(Model, Value):
if Value > Model:
return False
else:
return True
def simulation():
numberOfEvents = 0
numberOfTimesABCorrectlyCounteredC = 0
A = 60
B = 70
C = 80
while numberOfEvents < 10000:
AValue = random.randint(1,100)
BValue = random.randint(1,100)
CValue = random.randint(1,100)
APred = Predicts(A, AValue)
BPred = Predicts(B, BValue)
CPred = Predicts(C, CValue)
if APred == BPred and APred !=CPred:
numberOfEvents = numberOfEvents + 1
if CPred==False:
numberOfTimesABCorrectlyCounteredC = numberOfTimesABCorrectlyCounteredC + 1
return numberOfTimesABCorrectlyCounteredC
count = 0
number = 0
while count < 100:
number = number + simulation()
count = count + 1
print("After 1,000,000 events AB correctly countered C this many times:")
print(number)
Given models that predict a binary outcome (and not a probability), and the historical accuracy of those models based on a large pool of examples, is there a way to combine the models such that their overall predictive power is greater than any one individual model?
Side note: My stat-fu is not the strongest and I wouldn't be surprised if I get torched for something I missed (maybe in my tags of assumptions). No hard feelings, and either way, I appreciate your feedback.