4
$\begingroup$

So I was reading a lot of online articles using the chi-square test to determine the fairness of a die.

Correct me if I am wrong, but isn't the drawbacks of using a goodness of fit test is that you cannot conclude anything about individual sides of die, only if the die was fair or not?

Couldn't we solve this problem using binomial probability similar to a determining fairness of a coin but instead where we use a one-versus all approach? I know what I am writing below is kind of silly and rather pointless; I'm just trying to see if I truly understand the problem.

So instead of one Chi-Square test, we can do 6 tests and running a hypothesis test for each binomial distribution. Let's roll 100 times for each side (total of 600 times):

For side = 1:

$P(1) \approx .1666$

$P(2,3,4,5,6) \approx .8333$

$\mu = np = 100*.1666$

$\sigma^2 = npq = 100*.1666*.8333$

$...$

For side = 6:

$P(6) = .166$

$P(1,2,3,4,5) = .833$

$\mu = np = 100*.1666$

$\sigma^2 = npq = 100*.1666*.8333$

If any of the sides fall outside of the critical region, we reject the null hypothesis, and we can determine which side(s) is most likely biased.

Couple of questions:

  • Is this method even valid?
  • Would this result give us a similar result to a chi-square test?
  • Do we have to do any corrections due to the multiple comparisons problem?
Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ $E()$ should be $\Pr()$. $\endgroup$
    – user158565
    Nov 22, 2018 at 3:05
  • 1
    $\begingroup$ You would have to do some sort of multiple-testing correction to your significance level in order to maintain the same type 1 error rate, e.g. a Bonferroni correction. At that point, I believe you lower the power of your test if you want to maintain the same type 1 error rate. $\endgroup$
    – Joey F.
    Nov 22, 2018 at 5:32

3 Answers 3

Reset to default
5
$\begingroup$

I'm going to talk a little more in a general/intuitive sense since there are already great statistical resources out there for this. An example would be here.

The Chi-square test per se (the test statistic, p-value, conclusion) does not tell you about individual sides of the die.

However, you can look at the residuals (difference between observed counts vs. expected counts) to get an idea about individual sides of the die. Even better, you can use one standardized version of the residuals called the Pearson residuals. $$ r_{i} = \frac{O_{i}-E{i}}{\sqrt{E_{i}}} $$ for $i = 1$,...,$6$ sides of the die.

If $r_{i}$ is negative, then there were less observations of side $i$ than would be expected under the null hypothesis (of the die being fair), and if $r_{i}$ is positive, there were more observations than expected under the null.

Large (absolute) values of $r_{i}$ provide more evidence against the null hypothesis, and in fact all of the $r_{i}$ together contribute to your final chi-square statistic: $$ \chi^{2} = \sum_{i=1}^{6} r_{i}^{2} $$

You could perform significance tests of the $r_{i}$ individually as well. This would correspond directly to your technique - testing the individual probabilities of each the sides of the die separately. But then you have to ask what you are trying to answer. Are you interested in the whole die, or in just one side? That would determine if your method is valid.

If you're interested in just one side, you don't need to check the other sides. Just test that one side and ignore the individual tests and the overall chi-square test.

If your're interested in the whole die, then the method of checking each side individually will distort your Type 1 error rate compared to the chi-square test. To get an intuitive sense why, consider the formula above:
$$ \chi^{2} = \sum_{i=1}^{6} r_{i}^{2} $$

The chi-square test considers all of the residuals together. Under the null, some will be larger and some will be smaller, but the chi-square test will in effect average (in a broad sense of the term) all of their respective values. But individual tests hone in on the residuals with greatest (standardized) values. Overall, this would make your individual tests technique have higher Type 1 error rates than the chi-square test.

You could do multiple comparisons to guard against this higher Type 1 error rate. Based on your individual tests idea, you would want to guard against the family-wise error rate, so one appropriate multiple comparison technique would be the Bonferonni correction.

Improve this answer
$\endgroup$
4
$\begingroup$

I wonder if using confidence intervals for multinomial proportions might be a good approach. If the confidence interval for a side doesn't include the expected proportion of 1/6, that would be evidence that would flag that something is amiss with that side.

I don't know if there is a multiple testing problem with this approach.

Below I have some R code, with some hypothetical data. After 100 roles, Side 3 has a suspiciously high proportion. Confidence intervals for the multinomial proportions are generated with the Sison and Glaz method. These are also easy to visualize in a plot; here the horizontal line indicates the expected 0.167 proportion. Also included are a chi-square goodness-of-fit test and standardized residuals.

Install packages

if(!require(DescTools)){install.packages("DescTools")}
if(!require(ggplot2)){install.packages("ggplot2")}

Hypothetical data

Side = c(1,2,3,4,5,6)
Counts = c(15, 15, 26, 14, 16, 14)

sum(Counts)

    ### [1] 100

Sison and Glaz confidence intervals

library(DescTools)

MCI = MultinomCI(Counts)

MCI

   ###       est lwr.ci    upr.ci
   ### [1,] 0.15   0.06 0.2480369
   ### [2,] 0.15   0.06 0.2480369
   ### [3,] 0.26   0.17 0.3580369
   ### [4,] 0.14   0.05 0.2380369
   ### [5,] 0.16   0.07 0.2580369
   ### [6,] 0.14   0.05 0.2380369

   ### Here, est is the proportion for that side, 
   ### and lwr.ci and upr.ci indicate the confidence interval for that side.

Create plot

library(ggplot2)

Data = as.data.frame(MCI)

Data$Side = factor(Side)

Data$Proportion = Data$est

qplot(x = Side, y = Proportion, data = Data) +

geom_errorbar(aes(ymin = lwr.ci, ymax = upr.ci, width = 0.15)) + 

geom_hline(aes(yintercept=0.1667))

enter image description here

Interestingly, a chi square goodness of fit test doesn't indicate that the counts deviate from the expected values. I also ran an exact test (not shown), and the p-value there was 0.22.

chisq.test(Counts)

   ### Chi-squared test for given probabilities
   ### 
   ### X-squared = 6.44, df = 5, p-value = 0.2657

But a standardized residual > 2 suggests something is interesting with Side 3.

chisq.test(Counts)$residuals

    ### [1] -0.4082483 -0.4082483  2.2861904 -0.6531973 -0.1632993 -0.6531973

References: Goodness-of-Fit Tests for Nominal Variables. Caveat: I am the author of this webpage.

Improve this answer
$\endgroup$
3
$\begingroup$

As you have correctly perceived in your question, the main issue with the method you are proposing is that it involves multiple hypothesis tests to scrutinise a single underlying hypothesis, and it therefore suffers from the problem of multiple comparisons. As your question is currently framed, you are asking if your method is a valid test, and if it needs to be adjusted for multiple comparisons. However, at the moment you don't actually have "a test", you have six tests. If you want to combine this into a single test that is statistically sound, then yes, you will need to adjust for multiple comparisons.

At the moment you have not proposed a method to combine these six individual tests into a single overarching test, so as it stands, you do not yet have a fully specified testing method. However, if you were to proceed to form an overarching hypothesis test that combines these individual tests, you would need to combine the individual p-values of the tests into a single overarching p-value, with an appropriate adjustment for multiple comparisons. This would be tantamount to forming a single test where the test statistic is some function of the counts of outcomes. Since this is already what is done in a standard multinomial test, it is likely that any attempt to adjust your method to be statistically sound would lead to a method that is either similar to (or inferior to) some standard test.

If you would like to pursue this proposal in more detail, I would suggest some preliminary reading on existing multinomial tests used in categorical data analysis. I have set out a basic overview of the problem below, but I would recommend consulting some books on this to learn about the existing tests. If you feel that you could develop a competing test via your method, you would need to fully specify the method to aggregate the individual tests, and then derive the resulting statistical properties of the test, and compare it to existing methods.

Tests of fairness of a die: In the general case where you roll an $m$-sided die $n$ times, you will obtain face-counts that are distributed according to a multinomial distribution:

$$\mathbf{N} \sim \text{Multinomial}(n, \mathbf{p}),$$

where $\mathbf{p}=(p_1,...,p_m)$ is the vector of probabilities of the outcomes on the die. The die is fair if the probability vector is the uniform vector $\mathbf{u} = (\tfrac{1}{m},...,\tfrac{1}{m})$. There are a number of existing multinomial tests that can be used to test the hypotheses:

$$H_0: \mathbf{p} = \mathbf{u} \quad \quad \quad H_0: \mathbf{p} \neq \mathbf{u}.$$

Some of the available multinomial tests use the chi-squared distribution as an approximation of the large-sample distribution of the test statistic. If the data set is not too large it is sometimes possible to eschew the chi-squared calculation in favour of an exact test.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ +1 As some readers may only be consciously aware of one exact test (& so may think that test was being discussed) I felt I should point out that Ben's intent in mentioning 'an exact test' on the final sentence is to make a test whose actual significance level can be computed exactly, generally by basing it on the permutation distribution (under $H_0$) of whichever statistic is selected. (A brief discussion for the specific case of a Pearson goodness of fit chi-squared statistic is given here.) $\endgroup$
    – Glen_b
    Nov 22, 2018 at 22:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.