2
$\begingroup$

I have binomial outcome data for three groups that I am analyzing with Cochran's Q test. As discussed at this site, Cochran.

However, I have to repeat this seven different times because I am analyzing performance of three diagnostic tests to see if they can find resistance to seven different drugs.

When there are significant differences between the three groups, I perform a series of three paired post-hoc McNemar tests to further analyze the data.

However, my question is when there are not significant differences, do I still perform the three paired post-hoc tests ?

Part of me says why not, these McNemar's are valid tests. However, another part of me figures that if this is the case, why do the Cochran's Q test at all ?

Could some one point me in the correct direction ?

Thanks !

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Cochran's Q is so to speak "repeated-measures ANOVA" for dichotomous data, while McNemar's is like "paired t-test" for such data. So, normally one wouldn't do the second if the first is not significant and he decided to accept this. $\endgroup$
    – ttnphns
    Jul 15, 2014 at 16:10

1 Answer 1

Reset to default
2
$\begingroup$

Cochran's Q test is an analog to the repeated measures ANOVA for binary outcomes. As such it is an omnibus test with a null hypothesis that the population proportions behind each of the $k$ samples (three in your case) are all equal—H$_{0}\text{: }p_{1} = p_{2} = \dots = p_{k}$—with the alternative hypothesis that at least one population proportion is different than at least one other population proportion. Those who use such one-way omnibus tests typically proceed thus:

  1. Conduct the omnibus test;
  2. If you did not reject the omnibus test stop, you are done; otherwise:
  3. If you did reject the omnibus test conduct multiple post hoc pairwise tests to determine which pairs have different population proportions (McNemar's test is the appropriate test to use, although it is exactly equivalent to Cochran's Q for two groups, so you can you either for the post hoc tests); and
  4. If you are feeling extra competent, perform either family-wise error rate or false discovery rate multiple comparisons adjustments.

So how would that work out in your case?

  1. Conduct Cochran's Q test for your three groups.
  2. If you do not reject (1) stop you conclude you have no evidence that none of the population proportions behind your three samples are different from one another; otherwise:
  3. Conduct three pairwise tests (if you have groups A, B and C, these would be tests of A vs. B, A vs. C, and B vs. C). You can conduct these three tests using either McNemar's test or Cochran's Q for two groups;
  4. If you are feeling extra competent (and I just bet you are), you can perform a Bonferroni, Holm-Sidak, Benjamini-Hochberg or some other adjustment for multiple comparisons. At this point you may or may not determine that any numbers of these pairs are significantly different.
  5. Profit!

Why do people bother with the omnibus tests in the first place? Because laziness is a good thing, and the total number of possible tests, $m$, grows quite big as $k$ increases ($m$ can be as large as $\frac{k(k-1)}{2}$). Thus if we do not reject the omnibus test's null hypothesis we do not need to carry on with the arduous, achingly banal tedium of conducting more tests.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks Alexis! That makes sense to me. Laziness is not really a good reason (IMHO) because my coding does the testing for me.... I suppose another good reason to do omnibus testing is that if you set alpha =0.05 and you do 20 tests, you would expect at least one to be wrong even though you thought it was right. Omnibus testing allows you to do less testing, reducing that likelihood. $\endgroup$
    – user918967
    Jul 15, 2014 at 19:14
  • $\begingroup$ The usefulness of doing omnibus test followed by pairwise comparisons depends fully on the setting and the research question. There is no point in doing an omnibus test if the corresponding hypothesis is of no interest. A typical nonsense situation of doing such strategy is if you have one control and two experimental arms and you want to know which of the experimental groups to choose (if any). Then usually only the pairwise tests are of interest. $\endgroup$
    – Michael M
    Jul 15, 2014 at 21:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.