Reading the chi-square table

Question

For solving problems involving the chi-square distribution in my syllabus, we use this $\chi^2$ statistical table. However, I have a doubt in reading it.

Suppose there is an experiment where a die is thrown many times. We tabulate the times for which $1$ turns up, $2$ turns up and so on. Now we want to check whether the die is fair.

We can construct an expected values table, in which number of times each digit must show up will be $1/6$ times the total number of throws. Then we calculate $\chi^2=\sum\cfrac{(O_i-E_i)^2}{E_i}$ where $O_i$ and $E_i$ are the observed and the expected values at the $i^{th}$ trial respectively.

Then we consult the table. Degrees of freedom will be $\nu=6-1=5$. Now, my text says that the critical point of rejection is $p=0.05$. That means, if $\chi^2_{calc}<\chi^2_{table}(p=0.05)$, then my null hypothesis is false, and the die is fair.

But then, it also goes on to say that if $\chi^2_{calc}<\chi^2_{table}(p=0.95)$, then we would say that the die is more than $95\%$ fair. So, that must mean we are rejecting null hypothesis if we find that the die is just more than $5\%$ fair. Is that correct? I find it unusual that we would reject the null hypothesis only on $5\%$ confidence against it.

Answer 1

There is no such thing as "95% fair".

The fact is you cannot state with absolute certainty that the die is unbiased. The best you can do is say: if it's fair then it should give approximately-equal counts (low $\chi^2$ value); so if the $\chi^2$ value is too high then, with some quantified risk (in your case 5%) of being wrong, we will reject the hypothesis of fairness.

So it's a trade-off. You have to decide how to balance the risk of accepting a loaded die versus the risk of rejecting a fair one. In the real world, that could come down profit, reputation, whatever. Set the $\chi^2$ threshold too low and you'll reject too many fair dice; set it too high and you'll accept too many loaded ones.

If you do more trials (more rolls of the die), you'll get a cleaner separation between the behaviour of fair and loaded dice, and lower risk of a wrong decision. But again in the real world there would be a practical issue: is the improved ability to distinguish fair from loaded really worth the cost of rolling the die another however-many times?

Answer 2

That means, if $χ^2_\text{calc}<χ^2_\text{table}(p=0.05)$, then my null hypothesis is false, and the die is fair.

Not quite. It doesn't actually mean your null hypothesis is false. This is decision making under uncertainty. While you reject the null (concluding that it's false), your choice to do so can be wrong. In fact that "0.05" value tells you how often you'll reject the null hypothesis if it's actually true.

But then, it also goes on to say that if $χ^2_\text{calc}<χ^2_\text{table}(p=0.95)$, then we would say that the die is more than 95% fair.

This statement makes no sense to me. I'm not even sure what it's trying to say - what does "95% fair" mean? (Are you able to quote what they actually say?)

So, that must mean we are rejecting null hypothesis if we find that the die is just more than 5% fair. Is that correct?

No, it doesn't seem to say anything that makes sense; perhaps they mean to say that we should reject surprisingly small values of the test statistic.

We're trying to test the fairness of the die. For testing that, our rejection region only includes large values of the statistic (ones that indicate the die is unfair).

If it turned out the die rolled surprisingly close to fair, it would certainly not lead us to conclude the die was unfair.

If instead of suspecting the die was unfair, we thought that someone was "fudging the data" we might run a different test -- one where we check for low values of the statistic, but that wasn't the hypothesis we started with in this question, so we have no business rejecting the null when the results turned out to be really consistent with it. It's important to specify our rejection region before we see the data -- if we wanted to test for either an unfair die or someone fudging the numbers to make the die look really fair, we could do so, but we should not do so on the basis of discovering a low-chi-square after seeing the data.