1
$\begingroup$

I have a categorical variable for measured for two samples and I want to see if this variable differs significantly between the samples. I want to do a chi-squared test, with the samples as the columns and the categories of the variable as the rows (or vice versa). However, I have seen some people combining the two samples and then comparing one sample against this combined sample.

Is this correct? Are there any consequences of doing a sample A vs (sample A + sample B) comparison? My instinct is that you are less likely to find a significant difference between samples because you are comparing one sample against an average of the sample and the other.

EDIT: I'll provide an example illustrate my confusion. Here I run A vs B

m <- matrix(c(20, 10, 5, 10), nrow=2, dimnames=list(c("A", "B"), c("CatX", "CatY")))
m
  CatX CatY
A   20    5
B   10   10

chisq.test(m)

Pearson's Chi-squared test with Yates' continuity correction 
data:  m X-squared = 3.2512, df = 1, p-value = 0.07137

Then when I combine A and B

n <- matrix(c(20+10, 10, 5+10, 10), ncol=2, dimnames=list(c("A+B", "B"), c("CatX", "CatY")))
n
    CatX CatY
A+B   30   15
B     10   10
chisq.test(n)

Pearson's Chi-squared test with Yates' continuity correction 
data:  n X-squared = 0.99712, df = 1, p-value = 0.318

The p-value is much greater in (A+B) vs B in comparison to A vs B.

Reading the response to this question, I found this quote:

Thus, it comes out that chi-square tests the deviation of each of the two groups profiles from this average group profile, - which is equivalent to testing the groups' profiles difference from each other, which is the z-test of proportions.

If this is the case, then surely adding B to A decreases the difference between the two and therefore leads to larger p-values.

To clarify: if I want to test for independence between two categorical variables, is it ever suitable to compare one group verses a combination of the two? If so, what am I effectively doing by comparing one group to a combination of the two?

$\endgroup$
  • $\begingroup$ Logically, comparing A vs (A+B) is equivalent wrt conclusions with comparing A vs B, since A in both sides is the same. You may go on with what you were planning initially. For 2x2 frequency table the chi-square test of association is the same as the z-test of two independent proportions. $\endgroup$ – ttnphns Jul 14 '16 at 19:59
  • $\begingroup$ @ttnphns That comment strikes me as potentially misleading, because this isn't a question about logic but about statistics. Statistically, comparing A to A+B is very different from comparing A to B: the former compares strongly correlated data while the latter does not. Because of that, the $\chi^2$ test simply does not apply to the former comparison. $\endgroup$ – whuber Jul 15 '16 at 15:05
  • $\begingroup$ @whuber, when I was commenting the initial (short) draft of the Q, I didn't imply that the A vs (A+B) comparison be that same standard chi-square test that is applied to A vs B comparison. Neither I thought that the OP would be going that "straightforward" way of application. Instead, saying of "logic" I meant that A vs (A+B) comparison can be mathematically done (to account for the correlated samples) to produce the p-value of A vs B chi-sq test. Note that I didn't say "A vs C comparison where C=A+B" which would imply sample A "be lost in C". $\endgroup$ – ttnphns Jul 15 '16 at 15:37
  • $\begingroup$ (cont.) So, it is only after the OP edited the Q that my initial comment becomes "potentially misleading" from somebody's statistical point of view. Again to repeat: I didn't mean the same chi-square test to be applied in the two comparisons. I didn't mean just that when was saying go on with what you were planning initially. $\endgroup$ – ttnphns Jul 15 '16 at 15:44
  • $\begingroup$ If this is the case, then surely adding B to A decreases the difference between the two and therefore leads to larger p-values. tcn, you are totally correct observing this. The group A+B is contaminated with A, so A is compared with something which is partly A itself. Normally we would not do such things. $\endgroup$ – ttnphns Jul 15 '16 at 17:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.