Distinguish two distribution using chi-square test

Question

I have two dice, each with four faces.

For first one, each of the numbers 1,2,3,4 occur with 0.25 probability.

For the other, the faces have probabilities 0.4,0.3,0.28,0.02 respectively.

I take one die randomly and throw it 100 times. Now I want to know is it 1 die or die 2.

We can use Chi-square test to find this. Let outcomes are $n_1,n_2,n_3$ and $n_4$ respectively. Then I calculate $\chi^2=\frac{(n_1-25)^2}{25}+\frac{(n_2-25)^2}{25}+\frac{(n_3-25)^2}{25}+\frac{(n_4-25)^2}{25}$. If $\chi^2$ is less than some bound, it is from unbiased dice. Otherwise it is from biased dice. So in hypothesis testing

  1. H_0: Dice is unbiased
  2. H_1: Dice is biased

Is there any other test, which is better than this test? How to find type 1 and type 2 error? Instead of 100, if we increase number of throws we can bound both type 1 & type 2 error. How many throws are required so that both errors are less than 0.01?

Answer 1

Following @whuber's Comment: With the fair die, the distribution of the chi-squared statistic is approximately $\mathsf{Chisq}(\nu = 3).$ [Blue in figure.]

With the unfair die the chi-squared statistic is approximately noncentral chi-squared with $\nu = 3$ and noncentrality parameter as shown in the last section here. [Red in figure.]

For 100 rolls, the two distributions are sufficiently different to provide a good chance of accurate discrimination.

Answer 2

As whuber, Peter Flom and I all said (in different ways) in comments, a choice between two such options in the hypothesized circumstance isn't a standard hypothesis testing situation and that's not the best way to choose between them (though we can make use of the statistic -- see the note at the end).

The thing whuber was actually referring to (and which I alluded to previously as being an excellent way to decide between them) is as follows:

• Compute the likelihood ratio for the two options. It doesn't matter which one goes on the numerator, that just flips the direction of a decision rule.

• For a given set of observed counts in each category $(x_1,x_2,x_3,x_4)$, the likelihood for the biased die is proportional to $(0.4)^{x_1}\cdot(0.3)^{x_2}\cdot(0.28)^{x_3}\cdot(0.02)^{x_4}$ and for the unbiased die it is proportional to $(0.25)^{x_1}\cdot(0.25)^{x_2}\cdot(0.25)^{x_3}\cdot(0.25)^{x_4}$ (with the same constant of proportionality). The ratio is thereby $(1.6)^{x_1}\cdot(1.2)^{x_2}\cdot(1.12)^{x_3}\cdot(0.08)^{x_4}$

  and the log of the likelihood ratio is $d=\log(1.6)\cdot x_1 + \log(1.2) \cdot x_2 + \log(1.12) \cdot x_3 + \log(0.08)\cdot x_4$

  This will be positive when the unfair die has the higher likelihood and negative when the fair die has the higher likelihood. We can just adopt the naive decision rule of assigning a set of outcomes to the die with the higher likelihood. This is the same as saying "when $d$ is negative, decide in favor of the fair die; if it's positive decide in favor of the unfair die".

  (More generally we might want to shift the threshold a little way along from 0 to minimize the total cost - however at the large sample size we're looking at, the rate of the two kinds of errors are very similar, so if the costs of the two errors are the same we should be approximately at the optimum already.)

---

Let's look at its performance by sampling in exactly the fashion described in the question (choose a die at random, roll it 100 times) and then compute this log of the likelihood ratio criterion on the outcome; we can repeat that 1000 times.

As we see in the output there are two clusters of results, the orange ones in the left cluster, which are all the times a fair die was selected to generate the data, and the blue ones in the right cluster, which are all the times an unfair die was selected. At n=100, we don't see a single misclassified case. Indeed, if we do a million trials instead of 1000, we get only 33 misclassified cases:

table(data.frame(die=dres[1,],scoreneg=dres[2,]<0))
   scoreneg
die  FALSE   TRUE
  1     16 500339
  2 499628     17

---

Can we do something similar using a chi-square statistic?

Well, indeed, if we compute chi-squared statistics for each die (as if it were the null) and assign the outcome to the die which has the lower chi-squared statistic, we get quite good performance, though the misclassification rate is several times higher. On the other hand, we can improve it by moving the threshold a little (choosing the fair die only when the difference in the two chi-squareds $\chi^2_u-\chi^2_f$ is somewhat more negative).

As the sample size increases from 100, the performance of the two approaches (difference of log-likelihoods vs difference of chi-squared statistics) should become more and more similar.

---

Since someone nearly always asks, here's the R code I used to generate the histogram/rugplot above. I have not tried to make it pretty, nor understandable, it's simply what I used.

p1=rep(1,4)/4
p2=c(0.4,0.3,0.28,0.02)
pr=p2/p1
w=log(pr)
p=cbind(p1,p2)
dres=replicate(1000,{ch=sample(2,1);pr=p[,ch];  
       t=table(factor(sample(4,100,replace=TRUE,p=pr),levels=1:4));      
       c(ch,sum(w*t))})
hist(dres[2,],n=50, main="histogram of log-likelihood ratio")
t2=c(40,30,28,2)
t1=c(25,25,25,25)
abline(v=sum(w*t1),col=2)
abline(v=sum(w*t2),col=4)
abline(v=0,col=3)
rug(dres[2,dres[1,]==1],col="orange",pch="|")
rug(dres[2,dres[1,]==2],col="blue",pch="|")
abline(v=0,col=3)