Test if probabilities are statistically different?

Question

I completed a Monte Carlo simulation that consisted of one million ($10^6$) individual simulations. The simulation returns a variable, $p$, that can be either 1 or 0. I then weight the simulations based on predefined criteria and calculate the probability of $p$. I also calculate a risk ratio using $p$:

$$\text{Risk ratio} = P(p|\text{test case}) / P(p|\text{control case})$$

I had eight Monte Carlo runs, which consist of one control case and seven test cases.

I need to know if the probabilities of $p$ are statistically different compared to the other cases. I know I can use a multiple comparison test or nonparametric ANOVA to test individual variables, but how do I do this for probabilities?

---

For example are these two probabilities statistically different?:

Probabilities:

$P(p|\text{test #3}) = 4.08 \times 10^{-5}$

$P(p|\text{test #4}) = 6.10 \times 10^{-5}$

Risk Ratios:

$\text{Risk Ratio}(\text{test #3}) = 0.089$

$\text{Risk Ratio}(\text{test #4}) = 0.119$

Answer 1

If you have 1,000,000 independent "coin flips" that can produce 1 with probabilty (prob) and 0 with probability (1-prob), then the number of 1's observed will follow a Binomial distribution.

Tests of statistical significance are rejection tests, i.e. reject the hypothesis that the two parameters are equal if the probability that param2 is observed in test2 when the true value is param1 is less than a certain number, like 5%, 1%, or 0.1%. These tests are typically constructed from the cumulative distribution function.

The cumulative distribution function for a binomial is ugly, but can be found in R and probably some other statistics packages as well.

But the good news is that with 1,000,000 cases you don't need to do that.... you would if you had a relatively small number of cases.

Because you have 1,000,000 independent flips, the CDF of a normal distribution is a good approximation (the Law of Large Numbers plays a role here). The mean and variance you need to use are the obvious ones, and are in the Binomial Wikipedia article... You are then comparing two normally distributed variables and can use all the standard tests you would use with normally distributed variables.

For instance, if the true probability were 40*10^-6 then in 1,000,000 tests you would expect to see 40 +/- 6 positive cases. If the acceptance interval for a test is, for instance, 5 standard deviations wide on each side, then this would be compatible with both observations. If it were just 3 std dev wide on each side, one case would fit and the other would be statistically different.