Three people have independently developed models for predicting a coin flip. They take into account the launch angle, launch force, rate of spin, and various other factors to produce predictive models of varying quality. After a million flips:
- Model A predicts the result correctly 60% of the time
- Model B predicts the result correctly 70% of the time
- Model C predicts the result correctly 80% of the time
It seems to me that while C is clearly the best model, models A and B are not also worthless. I suspect that, if all three models point to the same result then that result is in actuality much more likely than any one individual model would suggest. Likewise I suspect that if Models A and B predict Heads while Model C predicts Tails, the actual probability of Tails is less than the historical 80% would suggest.
I ran a quick and dirty monte carlo simulation and after 5mil rounds found that, with surprising consistently, when Model A and Model B predicted Heads while Model C predicted Tails, Model C was correct only about 54% of the time.
import random def Predicts(Model, Value): if Value > Model: return False else: return True def simulation(): numberOfEvents = 0 numberOfTimesABCorrectlyCounteredC = 0 A = 60 B = 70 C = 80 while numberOfEvents < 10000: AValue = random.randint(1,100) BValue = random.randint(1,100) CValue = random.randint(1,100) APred = Predicts(A, AValue) BPred = Predicts(B, BValue) CPred = Predicts(C, CValue) if APred == BPred and APred !=CPred: numberOfEvents = numberOfEvents + 1 if CPred==False: numberOfTimesABCorrectlyCounteredC = numberOfTimesABCorrectlyCounteredC + 1 return numberOfTimesABCorrectlyCounteredC count = 0 number = 0 while count < 100: number = number + simulation() count = count + 1 print("After 1,000,000 events AB correctly countered C this many times:") print(number)
Given models that predict a binary outcome (and not a probability), and the historical accuracy of those models based on a large pool of examples, is there a way to combine the models such that their overall predictive power is greater than any one individual model?
Side note: My stat-fu is not the strongest and I wouldn't be surprised if I get torched for something I missed (maybe in my tags of assumptions). No hard feelings, and either way, I appreciate your feedback.