3
$\begingroup$

I have the following categorical data

  Control  Treatment
c1 285441     33296
c2 40637      4187
c3 737113     97433
c4 34036      3993

In other words, I have 2 multinomial distributions with 4 categories each. In effect, I would like to test to determine whether or not the treatment changes the distribution of category mix (c1,c2,c3,c4).

A quick glance at the data shows the following proportions for control and treatment respectively. For example, I calculate $p_{c1} = \frac{285441}{285441 + 40637 +737113 + 34036}$

Treatment (to 3 decimal places): $p_{c1} = .260, p_{c2} = .037, p_{c3} = .672, p_{c4} = .031$

Control (to 3 decimal places): $p_{t1} = .240, p_{t2} = .030, p_{t3} = .701, p_{t4} = .029$

Now it seems to me that, while there are some differences in the relative distribution of categories between control and treatment, this difference is not super drastic. So I'm going to run a chi-square test for homogeneity at significance level $\alpha = .0005$ (yes I know very small alpha). In our case, the degrees of freedom is 3, so we reject if we get a chi-square statistic $>17.7299$.

Under the null hypothesis, we expect the context and treatment to be the same. We calculate the MLE $\hat{p_1}$, which is the probability of landing in category 1. It is calculated as follows.

$\hat{p_{1}} = \frac{285441 + 33296}{\text{total count of both context and treatment}} \approx 0.258$

I'll calculate the first term of of my $\chi^2$ statistic (expected count for control in category 1), which I denote $E_{c1}$. The observation, denoted $O_{c1}$, is 285441. Hence, we have

$E_{c1} = \text{total count control} \times \hat{p_{1}} \approx 282919.389$

The first term of the chi-square statistic is given to be $ \frac{(E_{c1} - O_{c1})^2}{E_{c1}} = 22.475$

So from the first term alone, we are already in the rejection region. I redid my calculations and I compute my statistic (following the same approach above) to be $542.5772$. I don't trust my numbers, so I was hoping to verify that I am not misusing the chi-square test for homogeneity or making some idiot computational error. Hopefully, that clarifies the question more. Thanks!

$\endgroup$
  • $\begingroup$ Please add the self-study tag, read its tag-wiki. $\endgroup$ – Glen_b Jul 29 '15 at 10:20
  • $\begingroup$ Here's how to think on your feet about contingency tables: The square root of a cell's expectation is the likely amount of random variation in its value. (The counts will be approximately Poisson, so their standard deviations will be proportional to the square roots of their expectations. Your knowledge of the 68-95-99.7 rule tells you that a deviation from this expectation of more than 3 SDs will be rare.) E.g., the root of 282919 is 532. The count of 285441 is 285441-282919=2522 high, almost five SDs: it's too high. $\endgroup$ – whuber Jul 29 '15 at 15:59
2
$\begingroup$

They're clearly not homogeneous; the second row has a ratio of nearly 10:1, the third row only about 7.5:1. There's no need to test, with counts that large this will quite obviously be significant at any reasonable significance level.

However, I don't agree with the chi-square value you got. I calculated it "by hand" in R and got a much lower value than you give, and using chisq.test I got the same value (to about 5 figures). It should be a number quite a bit less than 1000.

If you want us to explain what you did wrong, show how you got that chi-squared value. What were your $E_i$ values? How did you calculate them?

Also, your significance level is wrong in two different ways at once; firstly, it should be small, not large* (I've already discussed this on another question of yours), and secondly, if the critical value really is 17.73, you calculated the significance level's complement incorrectly (leaving out a 9). The actual significance level you're using there is 0.0005 not 0.005.

*For a simple null, the significance level is the probability of rejecting the null hypothesis when it's true. You want to do that rarely, not often. Fortunately, your critical value is large, so you will do that rarely, but then you have to describe the significance level correctly; it's a small number.


Your first expected value calculation looks to be correct to the 3 figures you're working to (nice level of detail on that part, too; that helps us to spot errors). Also your new chi-square value matches mine to at least 3 figures. So unless I'm making the same errors you are*, you're doing it correctly.

*(we all make errors, and me as much as anyone, so it could happen)

$\endgroup$
  • 2
    $\begingroup$ +1 This answer is remarkable for its insight and thoroughness--you caught a lot of pitfalls that (I believe) most readers would at first sight have overlooked. (BTW, I upvoted the question, too, because it is unusual in providing an explicit context--helping you to figure out so much--and I don't think it should be faulted for including mistakes, but rather praised for making instructive ones.) $\endgroup$ – whuber Jul 29 '15 at 12:04
  • $\begingroup$ Hey. Thanks for the answer. I think I miscommunicated a few things so I clarified the question. Hopefully, my question will be more clear now. Would still very much appreciate your input. $\endgroup$ – Andy Lee Jul 29 '15 at 15:36
  • $\begingroup$ I updated my answer to respond to your update $\endgroup$ – Glen_b Jul 29 '15 at 21:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.