Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
4 teams play 2 games: football and basketball. (2 games just to better understand theory)
What should be the result so we can say that one team is stronger than other (statistically significant). I mean the winner is not due to luck.
Well, in order to say if there is a statistical significance you need to have some kind of distribution of the results, and this will depend on the "league". I.e. the distribution of results in the NBA is different from the one of the Youth League I used to play in :)
If you don't have a theoretical distribution for the results, you can try to use available data.
Let's assume you want to do so for NBA results. You can take the results of all NBA games (>4500) from 2014 and 2018 from Kaggle
Looking at the data, we can see that games finish on average with an 11 points win, with a standard deviation of 8.
If we do not make any assumption on the parametrical distribution of these results, but assume that the sample mean and variance represent well the population (which makes sense over 4500 samples), we can use Chebyshev's inequality:
$P(|X-\mu|>\sigma k) \le \frac{1}{k^2}$
Which, for a probability of 0.05 (usual p-value) gives us $k=4.47$. This equates to $\sigma k = 36$, which means that in order for a game difference to be statistically significant at 5 percent, the game needs to end with more than 36 points over the mean game (which is 11).
Summing up, the game would need to finish with at least 47 points difference!
(at 10% p-value, the difference would be 36 points at least).
Same calculations would apply to football or any other sport. Of course, you can also try to fit a parametric distribution to your game results, and you would have stricter limits, since Chebyshev is extremely general!
EDIT:
In order to take into account @D.W.'s comments, I tried only using games between more balanced teams. For example, I limited data to playoff games, to second round playoff games and after, or even to Conference Finals and after.
However, the distribution does not change that much. Even, it gets slightly more skewed!
On the other hand, I noticed that results - when computed in relative term - can be reasonably approximated with a gaussian (p-val < 0.01), and this can greatly improve the estimate compared to just using Chebyshev!
Therefore, a the relative win margin of the winning team comes from a truncated (as we only check the winning side) $N(0,0.0297)$
In this scenario, I would say that one team is statistically better than the other if the results end up in the tail, which means that (for 5%) the margin must be bigger than $1.96\sigma$.
A significant victory, would be one that has a relative margin of 0.339
For example, the 101:96 victory would have a margin of 0.052, which would not be significant. A victory of 129:96, on the other hand, would have had a 0.347 margin, which would have been statistically significant at 5%.
I am computing the margin as the score difference divided by the loser team's score.
I think it'll be difficult to perfectly model this without some domain knowledge, but a permutation test might provide a pretty decent approximation.
In general, a reasonable approach is to apply a hypothesis test. Here the null hypothesis is: the teams are evenly matched. Now to apply the null hypothesis testing methodology, we need to pick a test statistic, and we need to know the distribution of that test statistic for games where the null hypothesis is true.
I suggest using as the test statistic the relative margin of victory of your home team, as a proportion of the total number of points scored in the game. For your football game, the observed value of the test statistic would be $(2-1)/3 \approx 0.33$; for your basketball game, it is $(101-96)/197 \approx 0.025$.
Now we need a way to compute the distribution of this test statistic, for games where the null hypothesis holds. In other words, imagine simulating all ways that the games could have come out, under the assumption that the teams are perfectly matched, and calculate the distribution of the test statistic in all of those alternative universes. I suggest using a permutation test for this. In particular, I suggest you start with the assumption that the total number of points scored by both teams is constant, i.e., exactly 3 points are scored in all possible alternative outcomes for the football game, and exactly 197 points are scored in all possible outcomes for the basketball game (this is the part that is perhaps the most dubious, but we can come back to that).
In the football game, if both teams are evenly matched, then we're basically randomly choosing who scores each of the 3 goals. You can simulate this on a computer; in each simulated game, you randomly toss a fair coin three times and use each coinflip to determine who scored that goal, then compute the relative margin of victory for that game. Then you count the relative margin of victory for each simulated game, and plot a histogram of it. Or, in this case, we can analytically compute the distribution directly, as it takes the values $-1, -0.33, 0.33, 1$ with probabilities $1/8,3/8,3/8,1/8$, respectively.
Next, we compute the tail probability, i.e., the probability that a value chosen from this distribution has a value $\ge 0.33$. You can do that by simulation (counting the fraction of simulated outcomes where this happens). Or, you can do it analytically; it is $1/2$. So, we see that the p-value is $p=0.5$ for the football game. For the football game, this hypothesis test says that we cannot reject the null hypothesis; it seems plausible that this outcome could have happened by luck; it is not statistically significant at the $p=0.05$ threshold.
We can re-do the same computation with the basketball game. Everything applies as before; you can compute the distribution of the test statistic under the null using simulation, or analytically. I did that for you, and the answer is a probability of about $0.39$ that this distribution would yield a relative margin of victory that is $0.025$ or larger. So, for the football game, the p-value is about $p\approx 0.39$, so again we cannot reject the null; the outcome could have happened by pure luck; the margin is not statistically significant at the $p=0.05$ threshold. (However, if the score had been 111:86, a difference of 25 points, then the p-value computed in this way would have been $p \approx 0.043$, so the difference would have been statistically significant.)
The key modelling assumption I made here is that, for example, every possible match between those two football teams would have exactly 3 points scored. That is clearly a faulty assumption, as in a different universe there might have been more or fewer total points scored. It's plausible that this faulty assumption might not introduce too much bias, but it's not entirely clear. One way to try to get a somewhat more accurate model would be to capture data on the distribution of total number of points scored among all matches, and use that as a proxy for the total number of points that would have been scored in this match if we replayed it many times; it's an imperfect proxy, but better than just using 3.
For the basketball game, we have the same modelling problem, as well as another major flaw in the model: we assumed that each basket is worth 1 point, and that after each basket it is random which team scores the next basket. That's bogus, as in basketball each basket is worth 2 or 3 points, and there is a more complex structure (teams tend to alternate scoring attempts, and each attempt can succeed or fail: after one team tries to score a basket and succeeds or fails, typically the other team gets a chance to try). We could try to choose a more accurate model that would enable us to simulate possible outcomes of the basketball game. Or, we could somehow try to identify a set of historical games between teams that we somehow know to be evenly matched, and then use the distribution of the relative margin of victory among those games. It's not exactly clear how to do either of those with perfect fidelity, so it's likely that anything you do here will involve some modelling assumptions that are imperfect approximations to reality; hopefully you choose them in a way that does not introduce too much bias.
For these particular examples, the limitations in the modelling probably aren't significant enough that they would change the outcome of the analysis: it seems unlikely to me that a higher-fidelity model would reduce the p-value enough to cross the 0.05 threshold.
I hope this gives you some idea on how to think about the application of null hypothesis testing to your examples, as well as to understand their limitations and imperfections and challenges.
