How to test if two events with unknown probabilities are different or not

Let's say I have an "unfair die" with hundreds of possible results, and each result may have a different probability. However, I am interested only in two of these results and I want to test if they have the same probability or not.

Real data: After (at least) 500 rolls, the result A appeared 10 times and the result B appeared 44 times. How should I test this? And how should I involve the sample size? (I did at least 500 rolls, but they might even be 2000).

I've read about testing the fairness of a die using chi-square, but I don't know how to handle there the expected values, especially in relation to the number of times where the result was neither A or B.

It turns out the counts of the other outcomes don't matter: a suitable chi-squared test works just fine here.

Let the chance of outcome $$A$$ be $$p.$$ Under your null hypothesis, the chance of $$B$$ is the same and (therefore) the chance of seeing something other than $$A$$ and $$B$$ is $$1-2p.$$ Because your rolls are independent, probabilities multiply, implying the chance of what you observed is proportional to

$$L(p;10,44,500) = p^{10}p^{44}(1-2p)^{500-10-44} = p^{54}(1-2p)^{446}.$$

Upon taking logarithms and differentiating, it's easy to establish that this likelihood $$L$$ is uniquely maximized at the value $$p = \frac{1}{2} \frac{10 + 44}{500} = \frac{27}{500}.$$ The figure shows a plot of $$L(p);$$ the vertical axis is on a logarithmic scale.

Clearly, the expected counts under the null hypothesis are $$27$$ for both $$A$$ and $$B$$ and $$500-2\times 27 = 446$$ for all the others. Because the expected count for all the others is equal to the actual count, it contributes nothing to the chi-squared statistic:

$$\chi^2 = \frac{(10-27)^2}{27} + \frac{(44-27)^2}{27} + \frac{(446-446)^2}{446} = 2\frac{17^2}{27} \approx 21.4.$$

(This is identical to the value you would obtain if you ignored all the non-A, non-B outcomes and just worked with the counts of $$10$$ and $$44,$$ exactly as if you were flipping a potentially unfair coin and the outcomes A and B corresponded to its two sides: for a fair coin, you would guess that both counts should be around $$(10+44)/2=27$$ and thereby obtain the same value of chi-squared.)

A large value of chi-squared indicates a large deviation from what would be predicted by the null hypothesis. The chance of observing a statistic with a deviation at least this great is given by a chi-squared distribution. The particular one to use has one "degree of freedom" because one unknown value, $$p,$$ was involved in the likelihood $$L.$$ The "p value" given by this distribution is less than four in a million, which is so small you can safely conclude the null hypothesis is incorrect. In a technical paper you might write something like "the difference is significant ($$\chi^2(1) = 21.4,$$ $$p = 4\times 10^{-6}$$)."

Moreover, the evidence points to event $$A$$ having a smaller probability than $$B.$$

Finally, now that we know the two probabilities differ, we may estimate them (using maximum likelihood, as above, or otherwise) from the data. The maximum likelihood estimates are $$10/500$$ and $$44/500,$$ respectively.

Reference

In a post at https://stats.stackexchange.com/a/17148/919 I give the background and necessary conditions for applying a chi-squared test. You can verify all those conditions are satisfied here.

If the reasoning behind a null hypothesis and a p-value is unfamiliar, see my post at https://stats.stackexchange.com/a/130772/919 for an accessible account of this theory.

• Amazing answer! Thank you very much. How would the result change if there were 1,000 1,500 or 2,000 trials? I understand that the chi-squared and the p-values would be the same, and the maximum likelihood estimates would be 10/2000 and 44/2000, but... does it affect the power in any way? I also tried (in R) to redo the chi-squared test using a 50%-50% probability (equal prob A and B), 33%-67% (pB =2x PA, p-value=0.02363 ), 25%-75% (3x pv=NS), and 20%-80% (4x pv=NS). How does increasing the number of trials (without A or B) affect this? May 28 at 18:30
• I did not dwell on this issue because you can simply replace "500" by whatever your sample size is. That will change only the estimates of the probabilities, but not the test itself. Whether that changes the power depends on your perspective: if you define the power as a function of relative probabilities, the sample size will not affect it. If you try to define power in terms of absolute probabilities, you have a complicated analysis because the power depends on both probabilities.
– whuber
May 29 at 12:13
• In other words, we may think of the events being A=You toss an unfair coin and it shows heads, B=You toss the same unfair coin and it shows tails, C=You eat a banana. May 29 at 20:36
• @Hagen That's ambiguous, because the three events need to be mutually exclusive and "eating a banana" is not a certain event. A clearer model would be a three-sided die (that is, a trinomial distribution) with sides labeled "A", "B", and "neither." Now, for some intuition, interpret "neither" as "ignore this" or "roll again."
– whuber
May 30 at 13:10
• Thanks, I see. So... increasing the number of "neither" results will affect only the test measuring the "absolute probability" of A or B with respect of the whole die. However, the only thing that can increase the strength of the "ratio test" would be more successes for A and/or B. May 30 at 16:37

A Bayesian approach to the problem works out neatly. The reference posterior (see equation 6) of $$\phi\equiv{p_A/p_B}$$ is

$$\phi^{ref}\sim{}B'(\phi;x_A+1/2,x_B+1/2)$$,

where $$B'$$ is the beta prime distribution and $$x_A$$ and $$x_B$$ are the number of observations of $$A$$ and $$B$$ from the independent trials. Notice that the posterior distribution of $$\phi$$ is parameterized only by the counts of observations of $$A$$ and $$B$$, so the results of the analysis are the same whether the counts came from 500 rolls or 2000 rolls.

The beta prime CDF is the regularized incomplete beta function,

$$I_{\frac{\phi}{\phi+1}}(x_A+1/2,x_B+1/2)$$

The posterior likelihood that $$\phi>1$$ (i.e., $$p_A>p_B$$) is

$$1-I_{1/2}(10.5,44.5)\approx{7.97\times10^{-7}}$$

In R:

pbeta(0.5, 10.5, 44.5, lower.tail = FALSE)
#> [1] 7.972998e-07


As Bernardo and Ramón point out, the analysis works out similarly with the parameter $$\theta_{A|AB}\equiv\frac{\phi}{\phi+1}$$, which is the probability that the die shows $$A$$ given that the result of the roll is either $$A$$ or $$B$$. The reference/Jeffreys posterior of the probability of a Bernoulli trial is given by the beta distribution:

$$Be(\theta_{A|AB};x_A+1/2,x_B+1/2)$$