We are given a set of independent Bernoulli trials, each having separate claimed probability (which might not be accurate). We are also given a single observation for each of those events. How to estimate:
- Difference between claimed probability and observed probability
- Significance between observed probability and claimed probability (taking into account number of samples)
If each trial had exact same probability, we perhaps could've just computed mean and then a standard deviation, and compare that to the mean of the observation, but it's not clear to me if that would work for trials with different probabilities.
0.3, 0.5, 0.4, 0.85, 0.1
1, 0, 0, 1, 1
One approach I was thinking of is to use inverse probabilities minus one, then run a simulation to get distribution of expected sum (won't be Normal), then check at which percentile observed outcome sits, and take significance from that, but again, I'm not sure if that's a right approach, or how to incorporate a number of samples.
So with this approach it would be:
2.3333, 1.0000, 1.5000, 0.1765, 9.0000
2.3333, -1.0000, -1.0000, 0.1765, 9.0000
Expectation for an outcome is
1.9019607, which is
95.9% percentile. So it's as significant as having two standard deviations, hence significant.