# Distinguish two distribution using chi-square test

I have two dice, each with four faces.

For first one, each of the numbers 1,2,3,4 occur with 0.25 probability.

For the other, the faces have probabilities 0.4,0.3,0.28,0.02 respectively.

I take one die randomly and throw it 100 times. Now I want to know is it 1 die or die 2.

We can use Chi-square test to find this. Let outcomes are $$n_1,n_2,n_3$$ and $$n_4$$ respectively. Then I calculate $$\chi^2=\frac{(n_1-25)^2}{25}+\frac{(n_2-25)^2}{25}+\frac{(n_3-25)^2}{25}+\frac{(n_4-25)^2}{25}$$. If $$\chi^2$$ is less than some bound, it is from unbiased dice. Otherwise it is from biased dice. So in hypothesis testing

  1. H_0: Dice is unbiased
2. H_1: Dice is biased


Is there any other test, which is better than this test? How to find type 1 and type 2 error? Instead of 100, if we increase number of throws we can bound both type 1 & type 2 error. How many throws are required so that both errors are less than 0.01?

• Your question is unclear. How do you propose to use a chi-square test? There doesn't appear to be an identified null here, just two competing options -- as far as I can see, it's not a hypothesis testing situation at all, but something nearer to an estimation / classification / decision problem. Can you explain the circumstances? How does this question arise? .... I believe there's an excellent way to decide between them (which at n=100 will almost never go astray) but first the question needs to be more clearly formulated. Are you certain this needs to be done as a hypothesis test? Sep 20, 2019 at 11:14
• 1. You're now using the terms "biased dice" and "unbiased dice", which does at least hint at privileging one of the options but you haven't clearly identified the underlying problem you're trying to solve (in a way that would make it more clearly a testing problem rather than a classification problem). 2. Please identify the source of the problem. Is this an exercise for some class? Sep 20, 2019 at 12:44
• You can't use chi-square to solve this problem, at least, not in the way that you did. Your method includes no information about the non-uniform distribution. If you ran the same chi-square against that distribution, you might easily reject both nulls (or fail to reject both nulls). Sep 20, 2019 at 13:30
• Adopting one of the dice as a null hypothesis is artificial and asymmetric. Instead, use the multinomial likelihood ratio to decide which die is being cast.
– whuber
Sep 20, 2019 at 14:33
• @user12290 That is indeed the "excellent way to decide between them" I had in mind in my earlier comment. Sep 21, 2019 at 8:17

Following @whuber's Comment: With the fair die, the distribution of the chi-squared statistic is approximately $$\mathsf{Chisq}(\nu = 3).$$ [Blue in figure.]

With the unfair die the chi-squared statistic is approximately noncentral chi-squared with $$\nu = 3$$ and noncentrality parameter as shown in the last section here. [Red in figure.]

For 100 rolls, the two distributions are sufficiently different to provide a good chance of accurate discrimination. • Thanks a lot. But how to find the number of throws to bound the errors? Sep 20, 2019 at 17:16
• The noncentrality parameter is a multiple of the number of rolls. 100 seems pretty good. Somewhere between 11 and 12 may be a good critical value: in R, qchisq(.99, 3) returns 11.34487, and pchisq(11.34, 3, 31.52) returns 0.006632655. . Sep 20, 2019 at 17:51
• Thank yo so much. I get the idea now. Is it the best way to distinguish the two events? Sep 21, 2019 at 7:36

As whuber, Peter Flom and I all said (in different ways) in comments, a choice between two such options in the hypothesized circumstance isn't a standard hypothesis testing situation and that's not the best way to choose between them (though we can make use of the statistic -- see the note at the end).

The thing whuber was actually referring to (and which I alluded to previously as being an excellent way to decide between them) is as follows:

• Compute the likelihood ratio for the two options. It doesn't matter which one goes on the numerator, that just flips the direction of a decision rule.

• For a given set of observed counts in each category $$(x_1,x_2,x_3,x_4)$$, the likelihood for the biased die is proportional to $$(0.4)^{x_1}\cdot(0.3)^{x_2}\cdot(0.28)^{x_3}\cdot(0.02)^{x_4}$$ and for the unbiased die it is proportional to $$(0.25)^{x_1}\cdot(0.25)^{x_2}\cdot(0.25)^{x_3}\cdot(0.25)^{x_4}$$ (with the same constant of proportionality). The ratio is thereby $$(1.6)^{x_1}\cdot(1.2)^{x_2}\cdot(1.12)^{x_3}\cdot(0.08)^{x_4}$$

and the log of the likelihood ratio is $$d=\log(1.6)\cdot x_1 + \log(1.2) \cdot x_2 + \log(1.12) \cdot x_3 + \log(0.08)\cdot x_4$$

This will be positive when the unfair die has the higher likelihood and negative when the fair die has the higher likelihood. We can just adopt the naive decision rule of assigning a set of outcomes to the die with the higher likelihood. This is the same as saying "when $$d$$ is negative, decide in favor of the fair die; if it's positive decide in favor of the unfair die".

(More generally we might want to shift the threshold a little way along from 0 to minimize the total cost - however at the large sample size we're looking at, the rate of the two kinds of errors are very similar, so if the costs of the two errors are the same we should be approximately at the optimum already.)

Let's look at its performance by sampling in exactly the fashion described in the question (choose a die at random, roll it 100 times) and then compute this log of the likelihood ratio criterion on the outcome; we can repeat that 1000 times. As we see in the output there are two clusters of results, the orange ones in the left cluster, which are all the times a fair die was selected to generate the data, and the blue ones in the right cluster, which are all the times an unfair die was selected. At n=100, we don't see a single misclassified case. Indeed, if we do a million trials instead of 1000, we get only 33 misclassified cases:

table(data.frame(die=dres[1,],scoreneg=dres[2,]<0))
scoreneg
die  FALSE   TRUE
1     16 500339
2 499628     17


Can we do something similar using a chi-square statistic?

Well, indeed, if we compute chi-squared statistics for each die (as if it were the null) and assign the outcome to the die which has the lower chi-squared statistic, we get quite good performance, though the misclassification rate is several times higher. On the other hand, we can improve it by moving the threshold a little (choosing the fair die only when the difference in the two chi-squareds $$\chi^2_u-\chi^2_f$$ is somewhat more negative).

As the sample size increases from 100, the performance of the two approaches (difference of log-likelihoods vs difference of chi-squared statistics) should become more and more similar.

Since someone nearly always asks, here's the R code I used to generate the histogram/rugplot above. I have not tried to make it pretty, nor understandable, it's simply what I used.

p1=rep(1,4)/4
p2=c(0.4,0.3,0.28,0.02)
pr=p2/p1
w=log(pr)
p=cbind(p1,p2)
dres=replicate(1000,{ch=sample(2,1);pr=p[,ch];
t=table(factor(sample(4,100,replace=TRUE,p=pr),levels=1:4));
c(ch,sum(w*t))})
hist(dres[2,],n=50, main="histogram of log-likelihood ratio")
t2=c(40,30,28,2)
t1=c(25,25,25,25)
abline(v=sum(w*t1),col=2)
abline(v=sum(w*t2),col=4)
abline(v=0,col=3)
rug(dres[2,dres[1,]==1],col="orange",pch="|")
rug(dres[2,dres[1,]==2],col="blue",pch="|")
abline(v=0,col=3)

• Thank you so much. Can we find the distribution of d? Actually I want to find number of trials so that Type 1 error < 10^{-3} and Type 2 error < 2^{-30}. Sep 24, 2019 at 10:59
• I was not writing about a hypothesis test in my answer, because (as the problem is set up) a hypothesis test is the wrong tool. My answer opens with a discussion of that. If you're not doing a hypothesis test, you don't have type I and type II error; The way this is set up, you have misclassification probabilities. If you really do want it to work as a hypothesis test (why??), your problem setup needs to change (for example, your explicit specification of choosing between the two dice with equal probability is problematic) Sep 24, 2019 at 12:21