How to test if two events with unknown probabilities are different or not Let's say I have an "unfair die" with hundreds of possible results, and each result may have a different probability. However, I am interested only in two of these results and I want to test if they have the same probability or not.
Real data: After (at least) 500 rolls, the result A appeared 10 times and the result B appeared 44 times. How should I test this? And how should I involve the sample size? (I did at least 500 rolls, but they might even be 2000).
I've read about testing the fairness of a die using chi-square, but I don't know how to handle there the expected values, especially in relation to the number of times where the result was neither A or B.
 A: It turns out the counts of the other outcomes don't matter: a suitable chi-squared test works just fine here.
Let the chance of outcome $A$ be $p.$ Under your null hypothesis, the chance of $B$ is the same and (therefore) the chance of seeing something other than $A$ and $B$ is $1-2p.$  Because your rolls are independent, probabilities multiply, implying the chance of what you observed is proportional to
$$L(p;10,44,500) = p^{10}p^{44}(1-2p)^{500-10-44} = p^{54}(1-2p)^{446}.$$
Upon taking logarithms and differentiating, it's easy to establish that this likelihood $L$ is uniquely maximized at the value $$p = \frac{1}{2} \frac{10 + 44}{500} = \frac{27}{500}.$$  The figure shows a plot of $L(p);$ the vertical axis is on a logarithmic scale.

Clearly, the expected counts under the null hypothesis are $27$ for both $A$ and $B$ and $500-2\times 27 = 446$ for all the others.  Because the expected count for all the others is equal to the actual count, it contributes nothing to the chi-squared statistic:
$$\chi^2 = \frac{(10-27)^2}{27} + \frac{(44-27)^2}{27} + \frac{(446-446)^2}{446} = 2\frac{17^2}{27} \approx 21.4.$$
(This is identical to the value you would obtain if you ignored all the non-A, non-B outcomes and just worked with the counts of $10$ and $44,$ exactly as if you were flipping a potentially unfair coin and the outcomes A and B corresponded to its two sides: for a fair coin, you would guess that both counts should be around $(10+44)/2=27$ and thereby obtain the same value of chi-squared.)
A large value of chi-squared indicates a large deviation from what would be predicted by the null hypothesis.  The chance of observing a statistic with a deviation at least this great is given by a chi-squared distribution.  The particular one to use has one "degree of freedom" because one unknown value, $p,$ was involved in the likelihood $L.$  The "p value" given by this distribution is less than four in a million, which is so small you can safely conclude the null hypothesis is incorrect.  In a technical paper you might write something like "the difference is significant ($\chi^2(1) = 21.4,$ $p = 4\times 10^{-6}$)."
Moreover, the evidence points to event $A$ having a smaller probability than $B.$
Finally, now that we know the two probabilities differ, we may estimate them (using maximum likelihood, as above, or otherwise) from the data.  The maximum likelihood estimates are $10/500$ and $44/500,$ respectively.
Reference
In a post at https://stats.stackexchange.com/a/17148/919 I give the background and necessary conditions for applying a chi-squared test.  You can verify all those conditions are satisfied here.
If the reasoning behind a null hypothesis and a p-value is unfamiliar, see my post at https://stats.stackexchange.com/a/130772/919 for an accessible account of this theory.
A: A Bayesian approach to the problem works out neatly. The reference posterior (see equation 6) of $\phi\equiv{p_A/p_B}$ is
$\phi^{ref}\sim{}B'(\phi;x_A+1/2,x_B+1/2)$,
where $B'$ is the beta prime distribution and $x_A$ and $x_B$ are the number of observations of $A$ and $B$ from the independent trials. Notice that the posterior distribution of $\phi$ is parameterized only by the counts of observations of $A$ and $B$, so the results of the analysis are the same whether the counts came from 500 rolls or 2000 rolls.
The beta prime CDF is the regularized incomplete beta function,
$I_{\frac{\phi}{\phi+1}}(x_A+1/2,x_B+1/2)$
The posterior likelihood that $\phi>1$ (i.e., $p_A>p_B$) is
$1-I_{1/2}(10.5,44.5)\approx{7.97\times10^{-7}}$
In R:
pbeta(0.5, 10.5, 44.5, lower.tail = FALSE)
#> [1] 7.972998e-07


As Bernardo and Ramón point out, the analysis works out similarly with the parameter $\theta_{A|AB}\equiv\frac{\phi}{\phi+1}$, which is the probability that the die shows $A$ given that the result of the roll is either $A$ or $B$. The reference/Jeffreys posterior of the probability of a Bernoulli trial is given by the beta distribution:
$Be(\theta_{A|AB};x_A+1/2,x_B+1/2)$
