Ambiguity of chi-squared test of association; how can the conclusion change using the ''same'' sample?

Recently I have been struggling with the following two questions regarding the chi-squared test of independence / association during reading an article.

1. Is the chi-squared test able to detect no association between two groups among three at two levels? That is in a 3X2 contingency table.
2. Does the chi-squared test change its behavior with a larger sample that was otherwise perfectly proportional to a smaller sample?

I prepared a sample data set in R to seek the answers of these two questions. Though my initial discomfort about the chi-squared test have been proven, but over-acceptance of this test statistic in the scientific community have forced me to re-think. May be I have been missing something here.

# Question 1
x1 = c(100, 450, 450)
x2 = c(600, 450, 450)
d1 = rbind(x1,x2)
colnames(d1) = c("Type 1", "Type 2", "Type 3")
rownames(d1) = c("No", "Yes")
d1
chisq.test(d1)
# Significantly associated (not independent), though "Type 2" & "Type 3"
#  were not different at all!

# Question 2
d2 = d1/50
d2
chisq.test(d2)
# Not significant!!!

d3 = d2*2
d3
chisq.test(d3)
# Significant!!!


Is the chi-squared test able to detect no association between two groups among three at two levels?

That's not what the chi-squared test does at all. It is a test for any deviation from independence in the table (i.e. any form of association), and you have that; in a large sample those proportions are not consistent with independence.

--

If you want to test a subtable, test that subtable, rather than some larger table (but don't choose what you test based on looking at the data!). However, even if your sample has perfectly homogeneous proportions that doesn't tell you "no association"... because such things can easily happen -- even with large counts -- when the association is sufficiently mild.

how can the conclusion change using the "same" sample?

Because it isn't at all the same. The same proportions with different counts is telling you something different.

Consider a much simpler problem. I have a very pretty gold coin I want to use for the coin toss when I referee sports matches, but I worry because one side has a design that is raised more than the other side. So I want to check: is my coin fair?

Consider three cases:

1. I toss my coin three times and 2/3 of the time I get heads. Do I worry about my horribly biased coin?

2. I toss my coin six times and 2/3 of the time I get heads. Do I worry about my horribly biased coin?

3. I toss my coin three hundred times and 2/3 of the time I get heads. What about now?

In case 1, I have no reason to worry -- in fact with a fair coin the chance that the coin will come up 2/3 of the time (or worse) for one side ... is 100%

In case 2, I still have no reason to worry -- now the chance of a result at least that bad with a fair coin is over 68%; it happens often with a fair coin, so seeing it doesn't indicate unfairness.

In case 3, I sure as heck have reason to worry! The chance that one of the two sides comes up 2/3 of the time or more is 8 in a billion

The same proportions each time don't convey the same information, because the proportions associated with large counts are much more "precise" (the deviate much less from the population values).

Contingency tables have very similar properties; they're also count data, and their proportions become more precise as the count they're based on grows large. You can more readily distinguish differences as the total count grows, even though the proportions may be similar.

• I think all that was in my answer also ?
– user83346
Sep 5, 2015 at 13:15
• @fcoppens It might well be so. I didn't read your answer past the third paragraph where the introduction of matrices made me think that the level and form of the question called for a substantially more elementary answer. Sep 5, 2015 at 23:20
• no problem, it may be good to read the whole answer, but if we both say more or less the same then chances are high that it is more or less right what we say :-)
– user83346
Sep 6, 2015 at 7:21

I refer to this answer for background information on the $\chi^2$ test for contingency tables Chi-Square-Test: Why is the chi-squared test a one-tailed test?.

The test statistic is $X^2 = \sum_{i,j}\frac{(o_{ij}-e_{ij})^2}{e_{ij}}$,

in your case the $o_{ij}$ are the elements of your matrices $d1,d2,d3$. $e_{ij}=p_{i*} p_{*j} N$ where $p_{i*}$ and $p_{*j}$ are the (estimates of) the marginal probabeilities and $N=\sum_{i,j} o_{ij}$.

If we multiply $o_{ij}$ by a constant factor $k$ then we get $o_{ij}^{(n)}=k \cdot o_{ij}$. It is easy to show that $e_{ij}^{(n)}=k \cdot e_{ij}$, therefore

${X^{(n)}}^2 = \sum_{i,j}\frac{(o_{ij}^{(n)}-e_{ij}^{(n)})^2}{e_{ij}^{(n)}}=\frac{k^2}{k} X^2=k\cdot X^2$.

In other words after you multiply your matrix $d1$ with a constant, the test statistic $X^2$ is multiplied by the same constant ${X^{(n)}}^2 =kX^2$. This is what you see in R when you use the option correct=FALSE in chisq.test (no continuity correction, because the above formula is for the $\chi^2$ test without continuity correction):

# for d1 X2=267.8571
chisq.test(d1, correct=FALSE)

# for d1/50 X2=267.8571 /50
d2 = d1/50
d2
chisq.test(d2, correct=FALSE)


So the test statistic ${X^{(n)}}^2$ changes (is multiplied by $k$) , but the degrees of freedom of the test $(r-1)(c-1)=2$ (r is the number of rows, c the number of columns) has remained the same and therefore the p-value changes.

Is that normal ? Well what you do when you multiply the table with a factor $k$ is 'simulating' a change in sample size, and as the sample size increases, the results will become more significant:

#significant at 1pct level
chisq.test(d1, correct=FALSE)

#smaller sample size, insignificant at 1pct level
d<-d1/50
chisq.test(d, correct=FALSE)

# even worse, some cells have expected counts below 5 if sample too small and then problems with $\chi^$ approximation
d<-d1/100
chisq.test(d, correct=FALSE)


For the first question I point to the hypothesis that is tested in the $\chi^2$ test for contingency tables: $H_0: \text{ the row-variable is independent of the column variable }$ versus $H_1: \text{ the row-variable is dependent of the column variable }$.

I refer to the answer What follows if we fail to reject the null hypothesis?, to illustrate that 'statitistical evidence' can only be found for $H_1$, so to answer your first question, with the $\chi^2$ test you can find statitistical evidence that the row variable and the column variable are dependent (when you reject $H_0$) however, if you can not reject $H_0$ then you have to conclude that 'there is no evidence that they are dependent so we accept the assumption that they are independent'. So you can not 'detect' that there is 'no association'.

In your comments you say that $H_0$ is rejected (for $d1$) although there was no difference between type 2 and type 3. However, $\chi^2$ tests whether the row variable and the column variable are dependent. 'Type 2' and 'Type 3' are values that can be taken by the column variable.

So with the $\chi^2$ test based on your table $d1$ you conclude that the 'yes/no' answer depends on the variable 'type'. To find differences for the different values of the variable 'type' you will have to analyse the terms of $X^2$ namely $\frac{(o_{ij}-e_{ij})^2}{e_{ij}}$ or if you want the sign of 'deviation' then $\frac{(o_{ij}-e_{ij})}{\sqrt{e_{ij}}}$.

If you are sure that there is no difference between 'type 2' and 'type 3' then you can, or you should, work with a $2 \times 2$ table where the last two columns of $d1$ are aggregated.