An elementary question on binomial test: why should I take a sum?

Question

I conducted an experiment to compare two computer reversi programs, namely $A$ and $B$, in which I had them play with each other 50 times and $A$ won $B$ 31 times.

Now I would like to test if $A$ is stronger than $B$, i.e., the probability $p$ of $A$ winning $B$ (neither drawing or losing) is greater than 1/2, with significance level $\alpha$. According to Wikipedia's article on binomial test, I can compute $\frac{\binom{50}{31}+\dots+\binom{50}{50}}{2^{50}}$ and compare it with $\alpha$.

What I do not see is why I should take a sum, and consider the probability of winning 31, 32, ... or 50 times. If I were dealing with a continuous random variable, it would surely make sense to take a sum (that is, to integrate) to obtain probability from probability density, because there is no such thing as the probability of the variable coinciding exactly with the observed value. But I am dealing with discrete variable and I can calculate the probability of winning exactly 31 times.

So why should I take a sum (or do I really have to take a sum) ?

EDIT: I corrected my mistakes and clarified my question.

Answer 1

Max's comment answers your question: The definition of the $p$-value is the probability that you get a value at least as extreme, and this includes all outcomes more lopsided than the one you observed. It's your choice whether to consider $10-40$ more lopsided than $31-19$, whether to use a two-tailed test or one-tailed, but you must include $40-10$.

If you forget to include the more lopsided terms, then you will compute a small probability and automatically reject the null hypothesis when you use a large number of trials. If you play $1$ million games between equal opponents, the most likely outcome is an even split, and the probability of that is still quite small. ${1,000,000 \choose 500,000}/2^{1,000,000} \approx 1/(500\sqrt{2\pi}) \approx 0.000798 \lt 0.1\%.$ Every score has less than a $0.1\%$ chance to occur! So, if you don't add more extreme outcomes, you would simply confirm that some unlikely event had occurred just because there are a lot of possibilities when you have $1,000,000$ games. If you observe a score of $500,300-499,700$, the actual chance to see a score at least as lopsided in favor of $A$ when $A$ and $B$ are equal is $27.46\%$, and the probability of a score at least as lopsided in favor of either player is twice that, over $50\%$.

It is reasonable to ask why the $p$-value is defined this way. Whuber hinted that the Neyman-Pearson lemma is relevant. Another way to think about it is that we only want to have a chance $\alpha$ to reject the null hypothesis if the null hypothesis is true. If we have a linear ordering on how extreme outcomes are, and we define the $p$-value to be the probability of getting an outcome at least as extreme, then the event that we get an outcome with $p$-value lower than $\alpha$ has probability smaller than $\alpha$.

In different statistical procedures, there are times when you calculate just the probabilities of particular outcomes, such as a Bayesian update of a prior distribution. It's about $4$ times as likely to have a $31-19$ outcome if $A$ is actually a $60-40$ favorite rather than even, so you would strengthen your estimate of the probability that $A$ is a $60-40$ favorite by a factor of $4$ relative to the probability that $A$ and $B$ are even, not the ratio between the probabilities of observing events at least as extreme.