How to reduce type I and II error for determining bias of loaded dice for reduced legal costs?

Question

Suppose you are running a casino and that you are responsible for ensuring that all the dice are fair to avoid lawsuits. In order to do this, you might take a mean of 1000 throws of each die and perform a hypothesis test [using the central limit theorem, CLT] to see whether they are likely biased.

The average cost of a lawsuit is $£240,000$, whilst the cost of a die is $£3$, so in order to minimise costs would you aim to have $240000 \,\beta = 3\,\alpha$, where $\beta$ is Type II error, A.K.A., false negative rate, and $\alpha$ is the significance level of the hypothesis test (and also the probability of a Type I error, A.K.A., false positive rate)?

Now, in order to find the optimal $\alpha$ value, one must know the value of $\beta$, something that can only be calculated from die outcomes (which is what we are testing for in the first place), so the optimal solution cannot be found. That being said, however, I'm sure scenarios like this arise rather often, so how are they usually dealt with?

tldr: How would you find a threshold value for the mean of a die above (or below) which it should be considered biased whilst also keeping $240000\,\beta \text{ error} \approx 3\,\alpha \text{ error}$?

Edit: It seems my choice of example is rather poor, as a die shouldn't even be tested for fairness with a location test. That being said, however, my question more concerns the tradeoff between Type I and Type II error than the fact that a die is being used.

Answer 1

Concerning the Type I and Type II error comparison, a die roll of an unbiased die yields an outcome from a discrete uniform distribution. To test a candidate die for bias, one has to assume a type of bias distribution that is a discrete nonuniform distribution. Since there are several different ways of biasing a die, there is no unique distribution shape for a biased die. The method presented here is perfectly general in that it can be applied to any observable die bias (some bias types can be hidden), indeed, to any discrete problem type, and using binning, to some continuous distribution problems. Also, there are several allowed statistical tests for bias that could be compared for power for the biased die problem. None of these allow for the usual normal distribution assumption to be applied to dice. However, the method here is a data treatment for comparing a known distribution with an unknown one, such that $\beta$ cannot be assigned a priori.

A die has one of two official number patterns with opposite sides adding to 7; (1) the 1,2,3 vertex and the 4,5,6 vertex numbers increase clockwise, (2) numbers$-$same vertices$-$increase counterclockwise. This puts some constraints on the possible shapes for probability mass distribution for a biased die, but the remaining choices are hardly unique, i.e., there are many.

Cheating at craps in the US with loaded dice was common, usually by the casino, until the innovation of placing bets with the house as well as against it in 1907. However, even recently, many Las Vegas casinos' have been referred to their gaming commission for cheating with biased dice. In some jurisdictions, dice in current casino usage have security features including serial numbers.

Creative commons from https://boingboing.net/2015/07/31/dice.html.

However, in some locations at least, a casino may resist attempts to take any die home, unless it has been invalidated by mutilation.

Although it varies depending on the game and type of bet, the house take averages approximately 5%. I cannot do a full cost analysis without knowing who typically sues whom for what, how much it costs to record a number, $n$, of outcomes, how many dice are tested between lawsuits, what the savings would be for a particular level of exclusion of bias, and what level of bias is worth detecting. Depending on the particulars, both type I and II errors could have associated costs to either party in a lawsuit. As one will see, below, the confidence interval width of insignificantly biased dice proceeds as $\sim\dfrac{1}{\sqrt{n}}$, meaning that our cost for increased confidence for bias detection escalates as $\sim n^2$, when the cost of testing is $\sim n$.

Here are some methods of loading a die from the stack exchange gaming site

1) "Shaved" dice, which are not quite symmetrical, but slightly wider or narrower on one axis than on others. A shaved d6 with, say, the 1–6 axis longer than the others will roll those sides less often, making it "less swingy" than a fair d6 should be (but leaving the average roll unchanged). The name comes from cheaters actually shaving or sanding down dice to flatten them, but cheap dice may have this kind of bias simply due to being poorly made. Other similar biases due to asymmetric shape are also possible, especially in dice with many sides.
2) Uneven (concave / convex) faces may be more or less likely to "stick" to the table, favoring or disfavoring the opposite side. The precise effect may depend on the table material, and on how the dice are rolled. Again, cheap plastic dice case easily have this kind of bias, e.g. due to the plastic shrinking unevenly as it cools after molding. Uneven edges can also create bias, particularly if the edge is asymmetric (i.e. sharper on one side).
3) Actual "loaded" dice, i.e. dice with a center of gravity offset from their geometric center, may occur accidentally due to either bubbles trapped inside the plastic or, more commonly, simply due to the embossed numbers on the sides of the die affecting the balance. In fact, almost all dice, with the exception of high-quality casino dice deliberately balanced to avoid this kind of bias, will likely have it to some small extent.

Others

 4) Partially melting die in a microwave, 
 5) Variable loaded die having lead shot in a solid oil filled center that melts in hand heat. 

If I were loading a die myself, I might include a "now it's biased", "now it's unbiased" feature (e.g., using magnets).

A quick test is to float a die in salt water, where the concentration of salt is high enough to just float it, and give it a few flicks. If I were doing this myself, I would x-ray any opaque die. In a casino one can drop a die into a glass of water to see which side lands where, and players and dealers spin a die between two opposite vertices held between their fingers to see which side falls down. Also, so called "dice balancing calipers" are used to test for bias, and are available from gambling supply houses. Nothing will test for all possible methods of 'loading' a die. Keeping in mind that there is no simple test for human ingenuity, the most direct method of bias detection is to roll the die to see how it rolls. Under Las Vegas casino conditions, one would need an actual craps (or dice) table to do this, as casino dice have sharp edges and are not very useful for board games. In summary, one would not actually use statistics for a physical die bias testing, but on line casinos are another problem, and for those, there is no alternative to statistical testing.

One might think that the mean is a sensitive test choice, but not really because 3.5 is not an actual result. One can do sign testing on this, but this will be a lower power test, and like other location tests for would not detect numerically balanced biases. Fisher's exact test would be more interesting and possibly more powerful. If I assume that ordinary t-testing can be used, I will be violating assumptions as a discrete uniform distribution is not the same as a normal distribution. If I persist in doing this, the p-values resulting will be non-normally distributed, e.g., a very approximately, a Laplace distribution (it may be a generalized normal distribution, do not know exactly).

(Image of $t$-test probabilities from 10,000 simulated rolls.) What exactly this is is not that interesting because a disadvantage of using the mean value is that it may not detect certain methods of biasing the results. For example, I could make a die so that it would be equally difficult to land on a one and a six, and the average would be unchanged (e.g., shaved die). So, we need something that tests for each of the six number "cells" simultaneously. Chi-square testing is just such a test.

Chi-squared distribution use for die roll testing is routine. This tests the outcome of each result against its expectation for each possible outcome with the least amount of fuss. And a common rule of thumb is to have at least five times as many rolls as there are sides on the die.

The derivation of Pearson Chi-squared from the central limit theorem appears here and press the [show] link. An example for d6 testing appears here. An analysis of a simulation in Mathematica of 10,000 rolls (die) with testing appears next.

 nt = 10000; die = 
  RandomVariate[DiscreteUniformDistribution[{1, 6}], nt]; h = 
  Histogram[die, Automatic, ChartStyle -> {Opacity[0.3]}];

 Print["The number of 1's through 6's for 10000 simulated die rolls is ", 
  bc = BinCounts[die, {1, 7}], " having Chi-squared p = ", 
  bcdie = BinCounts[die, {1, 7}]; s = Sum[(nt/6 - bc[[j]])^2/(nt/6), {j, 1, 6}];
  p = 1 - N[CDF[ChiSquareDistribution[5],s]], 
  ". The Bonferroni corrected Poisson distribution confidence interval is from ", 
  low = InverseCDF[PoissonDistribution[10000/6], 0.025/6], " to ", 
  high = InverseCDF[PoissonDistribution[10000/6], 1 - 0.025/6]]

 lh = Plot[{low, high}, {x, 1, 7}, 
 PlotRange -> {{1, 7}, {0, Automatic}}];
 Show[lh, h]

The number of 1's through 6's for 10000 simulated die rolls is {1679,1696,1619,1651,1689,1666} having Chi-squared p = 0.78885. The Bonferroni corrected Poisson distribution confidence interval is from 1560 to 1775.

The confidence intervals from 0.4167% to 98.583% for individual cells are equivalent to 95% CI's for every cell. This amounts to a $+6.5$% and $-6.4$% spread using a Poisson counting model, suggesting that, a $12.9$% total range around 1/6th of the number of rolls would not detect a loaded die on all faces, and any die face showing a count beyond that would suggest significant bias.

Now the number ($n$) of rolls we need to perform to reduce our 95% confidence interval to a given fraction $y\sim\dfrac{12.8173}{n^{0.499151}}$ as per the following plot,

is a power function suspiciously decreasing proportional to approx. $1/\sqrt{n}$. Note that the Poisson confidence interval itself, in blue, ratchets for low count rates, not unexpectedly as Poisson counting is a whole number result.

Calculation via simulation of Type I error (false positive rate, FPR) and Type II error (false negative rate, FNR) requires 1) models for biased and unbiased dice 2) varying amounts of bias 3) selection of alpha 4) multiple random outcomes to build up a table.

The model used was to simulate extra weight placed by drilling holes under the dots of the six to make two bias models, one for which the six shows 4/10$^{\text{ths}}$ as frequently a one or six result (a $\frac{3}{2}$ odds ratio of seeing a one versus a six) and one for 1/3$^{\text{rd}}$ of the time (an odds ratio of twice). As the six is opposite the one, we should roll a one 6/10$^{\text{ths}}$ of the time, and 2/3$^{\text{rds}}$ of the time, that we see either number, respectively.

The biased die on the left panel is one trial of 1000 rolls of the die with a 1/3$^{\text{rd}}$ bias. The 95% and 99% confidence intervals are shown which represent the choices of $\alpha$ of 0.05 and 0.01 respectively. We then repeat this experiment 1000 times to accumulate histogram information, which is next displayed as probability histograms.

This shows the biased die results in orange and the unbiased die results in blue. The histogram categories are conveniently displayed at 0.05 and 0.01 probability widths and scaled for probability so that we can read power (1-FNR = 1 - type II error, in orange) and type I error (FPR, in blue) from the result in the first histogram category. Doing this repeatedly allows us to build up a table for 1000 trials of each level of bias for each type of error and each $\alpha$, shown next. This is done using the Chi-square distribution calculation, as it is more accurate than Poisson for this problem.

$$\left( \begin{array}{ccccc} \alpha & \text{Bias} & \text{n} & \text{FPR} & \text{FNR} \\ 0.05 & \text{4/10} & 200 & 0.054 & 0.797 \\ 0.05 & \text{4/10} & 1000 & 0.043 & 0.199 \\ 0.05 & \text{4/10} & 4000 & 0.052 & 0. \\ 0.01 & \text{4/10} & 200 & 0.008 & 0.918 \\ 0.01 & \text{4/10} & 1000 & 0.008 & 0.375 \\ 0.01 & \text{4/10} & 4000 & 0.019 & 0. \\ 0.05 & \text{1/3} & 200 & 0.039 & 0.486 \\ 0.05 & \text{1/3} & 1000 & 0.044 & 0.001 \\ 0.05 & \text{1/3} & 4000 & 0.045 & 0. \\ 0.01 & \text{1/3} & 200 & 0.011 & 0.728 \\ 0.01 & \text{1/3} & 1000 & 0.017 & 0.004 \\ 0.01 & \text{1/3} & 4000 & 0.012 & 0. \\ \end{array} \right)$$

Notice that the recovered type I errors are on the same order as those specified a priori, and are fairly accurate. Most notably, there is a variable not specified in the question. That is, the more subtle the bias, the more difficult it is to detect, i.e., the power then decreases.

Specifying a given ratio of type II and $\alpha$, as is requested in the question is ambiguous as it does not relate to accuracy of testing. What is really desired is high accuracy of testing, i.e., $\text{Accuracy} =1-(\text{FPR}+\text{FNR})$, which is achieved by minimizing the sum $(\text{FPR}+\text{FNR})$.

The above method is general, and applies to electronic results with a single die outcome of bias, like an all-too-frequent one, something that is not possible with a physical die which we could not bias for one's without reducing the frequency of outcomes of six's.

My advice is, if you want to gamble, buy shares.

Answer 2

I'm not sure I'd approach the problem this way. First, you have two hypotheses: the die is fair or not fair. If it is not fair, the distribution of throws will not be uniform (taking a mean of the throws is not an efficient way to measure this). Rather, I'd record the distribution of throws and after a few dozen, start calculating the empirical probability distributions (perhaps by chi-square or by way of binomial distribution depending on the game being played).

Factoring in the cost of a lawsuit doesn't enter into the equation for me, because the die should be eliminated regardless as soon as you're certain it's not fair. That said, there are applications of the sequential ratio probability test (SPRT) that are geared towards this. E.g., detecting manufacturing anomalies based on a sequential sample of product.

Answer 3

TLDR – the problem is underdefined

Aim

The details provided in the question can only lead to a relative balance of errors. Simply rearrange the equation and $ P(TypeII Error) ≈ 3P(TypeI Error)/240000$.

This is not what is wanted or needed, rather it is to know what threshold for mean that would be used as a cut off. I.e. what effect size minimises the cost function of your situation.

The cost of a typeI and typeII error have been defined, but not of the test to determine these.

Why does this matter?

The effect size that you can reliably measure depends on the acceptable type I error and the acceptable type II error. Below is a typical sample size calculator ( Box 1 in https://academic.oup.com/ndt/article/25/5/1388/1842407, open access) (OP indicated a generic answer was preferred so I have used a formula for normal distributions as this s probably more widely used, the principles are the same for other distributions types) $$n = 2*((a + b)^2δ^2)/(μ_1−μ_2)$$where a is the z multiplier to achieve the desired $\alpha$ level (1.96 for $\alpha$ = 0.05), while b is the equivalent for $\beta$. $\sigma$ is the standard deviation of the test result and $(\mu_1-\mu_2)$ is the effect sizem, we can simplify this to the threshold since an unbiased average would be 0 so we can drop $\mu_2$). $n$ in this specific example is the number of repeated trials to be carried out on an individual die. Rearrange this to solve for the effect size: $$ \mu_1 = 2*((a + b)^2δ^2)/(n)$$ What does this equation tell us? The issue of type I and type II error trade-off only applies when sample size and effect size are set to a constant. You can push both $alpha$ and $beta$ down ever lower and detect the same effect size by increasing the number of repetitions. But repeating a test increases the cost.

Since your costs link type I and II errors you can replace $a+b$ with $Z_\alpha + Z_{3\alpha/240000\alpha}$ Then you will need to either:

1) define an upper limit on what you are willing to pay for testing the dice, this will then leave you with two variables to solve for (effect size and alpha)

2) solve for repetitions, alpha and effect size simultaneously

You either need to define even more limits up front or use some post-hoc decision criteria to decide which balance of the possible outcomes is best suited.

If this were a reflection of a real world example, you would want to do a more detailed cost/benefit assessment (non-comprehensive list).

1. The cost of repetitions for your measure of fairness (this will guide your sample number, if it is say £1 per 10 reps, then it will cost 33x the cost of the dice to do 1000 reps).

2. The cost of failing to detect a bias (you give this as £240000 in you example for legal expenses, but in the real world the loss of earnings due to reputation damage and many others may be worth throwing in)

3. The cost of rejecting a fair dice (given as £3 in this case, in more generic cases there may be other issues to consider, including revenue boost due to enhanced reputation)

4. If you want estimates of absolute costs and risks you would need to define absolute values for how the scenario would be used in the real world. E.g. how many dice do you envisage deploying in the real world and how many times will each be used?

A useful reference: https://www.graphpad.com/guides/prism/7/statistics/stat_sample_size_for_which_values_o.htm?toc=0&printWindow

Answer 4

As I understand, the core part of the question is "how to choose $\alpha$ and/or $\beta$"?

Maybe it's me, that is (non-statistically) biased to interpret the question like this. But recently I had numerous discussions and thoughts about this very question. In a nutshell: there is no general rule, it depends on each and every case.

Textbook examples usually contain an outside element, e.g. what kind of error is "morally"/"socially"/"politically"/etc. more or less acceptable. In your case it might be "financially". Examples are:

In law/court, what is worse: that a criminal walks free (type A) or that an innocent person goes is sentenced (type B).

In medicine, a test: gives a false positive -- a healthy person is tested as sick or a false negative -- a sick person tests as healthy.

In any case, the discussion becomes something entirely non-statistical, non-mathematic. In the somewhat unusual/contrived/complicated example the additional step involves the calculation of expectation values and the minimization of costs/risks in light of the interpretation.

I somewhat digress, but it's similar to the "risk management" in a famous movie, that is intended to offend: a car company calculates the cost of a call-back for repairs versus the cost of a lawsuit and ensuing damages for accidents that are caused by the particular failure. The second calculation is based entirely on statistics, but the decision rule surely isn't.

In the case of the casino, it's simply regulated. There are no lawsuits for "unfair" dice. And in the case of a truly unfair die, the plaintiff would have to prove it that it is the casinos' fault. In online casinos the inverse is true; there are departments for fraud detection.