How to graphically show how "normal" a collection of dice rolls are in a game

I was trying to think of a way to show how "normal" a set of dice rolls are in a game like catan. In Catan, each turn you roll 2d6 and the sum of those 2d6 dictate game outcomes. A game can last anywhere from 20 - 50+ turns.

For examples sake, let's say we roll 36 times in a game. For a perfectly "normal" game the dice sum 2 would roll exactly once, as would 12. The dice sum 7 would roll 6 times. This forms a triangular distribution. In the distribution of all possible sets of dice results, this would be exactly in the middle of the distribution.

One way to do this might be to look at the total sum of all the dice sums (2d6) throughout the game. At the lower extrema someone could roll 2 36 times (so 72) and at the highest extrema someone could roll 12 36 times (432). The average then would be 7 * 36 = 252. So you'd have a distribution of sum results with around 252 with a min of 72 and max of 252. Is the distribution also triangular? I think so.

However, this isn't a very good score of how "normal" a game might be, because there is a big difference between rolling specific numbers. For example if no 7s were rolled the entire game this would be very abnormal and greatly affect gameplay.

So looking at the sum isn't a very good score.

I then thought about trying to compute the Z score for each result and somehow combine them. We know the expected frequencies of each 2d6 result in the totals number of rolls. We can then take the actual frequency and divide by the standard deviation for each roll to get this.

So (E(2d6r) - O(2d6r)) / Std(2d6r) where 2d6r is a specific result on 2d6 (like 12) and E is expected, O is observed and Std is standard deviation.

At this point we could sum up the absolute values of these z scores and get a total score. If the distribution matched exactly this score would be 0. In the most extreme case of rolling only 2s or 12s this number would be the largest it could be. How is this result distributed? Obviously it's going to be right skewed but it is normally distributed where I can say x% of results are less than some positive X value?

Is this z score calculation the most statistically appropriate for determining the "normal"-ness of a specific set of results given that we care about the frequencies of each results and not just the sum?

• You are asking how to test the goodness of fit of a distribution. A standard and (IMHO) very effective method for discrete distributions with small support, like yours, is the chi-squared test: see stats.stackexchange.com/search?q=chi+squared+fit+distribution. Z-score calculations won't accomplish anything for you.
– whuber
Oct 17, 2023 at 18:45

With games using dice it's important to pay attention to how the game mechanics relate to what matters in the game play. For example (speaking outside specifics of Catan) in a game where you're just trying to roll a total that was at least some fixed target, only the proportion of times you do that matters; any biases that don't affect the outcome -- like more 2's and fewer 3's than expected when the target is 8 -- would be irrelevant. On the other hand if the target varied you might consider displaying the whole cdf. Which is to say, you may well get a different answer about a suitable choice for different games.

In Catan, each individual outcome is typically important. I concur with whuber's comment that the chi-squared test would be a suitable test for this case, but it's not a display. A warning -- at these sample sizes, the dice would have to be pretty far from fair to show up anything on a chi-squared test with at most 50 rolls.

I will discuss a potential display I'd be inclined to look at. [I may come back and add another but I want to think about advice relating to some consequences of discreteness for that one.]

You could plot the proportion of times each outcome was rolled, with the expected proportions for a fair die (so far, it is straightforward). Here's an example for a game with 36 rolls.

Of course this doesn't show how unusual what we see is; we could add an interval around the expected proportions to indicate a range of observed probabilities consistent with a fair pair of dice. Alternatively to that interval you could put a confidence interval around each observed proportion. The coverage of such a set of individual intervals would need to consider the fact that you're looking at many such comparisons simultaneously -- the chance at least one would be outside such an interval is much higher than the individual ones. You might well want to look at a high individual coverage, or consider two different intervals.

At the same time, you could not use the asymptotic distribution of the counts to get the interval here, because $$n$$ is quite small (e.g. a game of 25 turns may very easily see no 2's or 3's, and the probability you don't see a "2" say is quite high, around $$50\%$$), so choice of what interval to display is not trivial. See the discussion at https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval for example. Keep in mind that when choosing a higher coverage, the discreteness problem is worse.

Even in the middle of the distribution, counts that are too low are only likely to "show up" at relatively low coverage probabilities (e.g. at n=25, even a 95% interval for each might not include "$$0$$" for just the middle three values), which means high probability of at least some values being outside overall.

Consequently, with so few rolls, this sort of thing would tend to only have much hope of finding individual errors "on the high side". It would tend to find the problem with my Catan dice easily; they roll "7" roughly 1/3 of the time, across many dozens of games (and games tend to be on the longer side as a result, helping show the bias more clearly).

Even with that caveat in mind, a similar warning to the one for the chi-squared test - about not seeing anything with such a small sample - applies here. Your power to see anything not explainable as just random variation is quite low.

If the number of rolls was higher -- across ten games, say -- it would be a more useful display, and you would begin to detect differences that matter.

• +1. But for a really effective display I would consider a rootogram or at least some indication of the standard errors of the proportions.
– whuber
Oct 18, 2023 at 16:04
• The standard errors would certainly be straightforward but I feared they might tend to be used to implicitly construct the asymptotic intervals I sought to avoid given the small samples combined with small probabilities. When combined with some additional information, they might work better. Oct 18, 2023 at 23:58