We are given a set of independent Bernoulli trials, each having separate claimed probability (which might not be accurate). We are also given a single observation for each of those events. How to estimate:

  1. Difference between claimed probability and observed probability
  2. Significance between observed probability and claimed probability (taking into account number of samples)

If each trial had exact same probability, we perhaps could've just computed mean and then a standard deviation, and compare that to the mean of the observation, but it's not clear to me if that would work for trials with different probabilities.

Probabilities: 0.3, 0.5, 0.4, 0.85, 0.1

Outcomes: 1, 0, 0, 1, 1

One approach I was thinking of is to use inverse probabilities minus one, then run a simulation to get distribution of expected sum (won't be Normal), then check at which percentile observed outcome sits, and take significance from that, but again, I'm not sure if that's a right approach, or how to incorporate a number of samples.

So with this approach it would be:

Random Variables: 2.3333, 1.0000, 1.5000, 0.1765, 9.0000

Outcomes: 2.3333, -1.0000, -1.0000, 0.1765, 9.0000

Expectation for an outcome is 1.9019607, which is 95.9% percentile. So it's as significant as having two standard deviations, hence significant.

The distribution looks as following: enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.