20
$\begingroup$

I'm conducting a chi-square goodness-of-fit (GOF) test with three categories and specifically want to test the null that the population proportions in each category are equal (i.e., the proportion is 1/3 in each group):

                OBSERVED DATA
Group 1     Group 2     Group 3     Total
  686              928            1012        2626

Thus, for this GOF test, the expected counts are 2626(1/3) = 875.333 and the test yields a highly-significant p-value of < 0.0001.

Now, it's obvious Group 1 is significantly different from 2 and 3, and it's unlikely that 2 and 3 are significantly different. However, if I did want to test all of these formally and be able to provide a p-value for each case, what would be the appropriate method?

I've searched all over online and it seems there are differing opinions, but with no formal documentation. I'm wondering if there is a text or peer-reviewed paper that addresses this.

What seems reasonable to me is, in light of the significant overall test, to do z-tests for the difference in each pair of proportions, possibly with a correction to the $\alpha$ value (maybe Bonferroni, e.g.).

Improve this question
$\endgroup$
12
  • $\begingroup$ t-tests would not be suitable. You could do pairwise goodness of fit tests (proportions tests). What differing opinions did you find? $\endgroup$
    – Glen_b
    Jan 27, 2015 at 0:11
  • $\begingroup$ Sorry - I meant z-test (for difference in two proportions). I'll edit. $\endgroup$
    – Meg
    Jan 27, 2015 at 0:16
  • $\begingroup$ This link says to group all other groups vs. the one of interest (it's for the Fisher's exact test, but this link is redirected from another link about the chi-square, where the author says to apply the same method for the chi-square as for the Fisher's exact): biostathandbook.com/exactgof.html#posthoc But this isn't really what I want - I want pairwise, not one group against all others. $\endgroup$
    – Meg
    Jan 27, 2015 at 0:22
  • 2
    $\begingroup$ Most other sources I find talk about a contingency table setting, not a GOF test. $\endgroup$
    – Meg
    Jan 27, 2015 at 0:28
  • 1
    $\begingroup$ Yes, you could do proportions tests (whether done as one-sample z-test or binomial test, or chi-square test) of each pairwise comparison. You have no need to do one-vs-all comparisons. $\endgroup$
    – Glen_b
    Jan 27, 2015 at 0:34

2 Answers 2

Reset to default
12
$\begingroup$

To my surprise a couple of searches didn't seem to turn up prior discussion of post hoc for goodness of fit; I expect there's probably one here somewhere, but since I can't locate it easily, I think it's reasonable to turn my comments into an answer, so that people can at least find this one using the same search terms I just used.

The pairwise comparisons you seek to do (conditional on only comparing the two groups involved) are sensible.

This amounts to taking group pairs and testing whether the proportion in one of the groups differs from 1/2 (a one-sample proportions test). This - as you suggest - can be done as a z-test (though binomial test and chi-square goodness of fit would also work).

Many of the usual approaches to dealing with the overall type I error rate should work here (including Bonferroni -- along with the usual issues that can come with it).

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thank you for your advice and for posting as an answer. I, too, was a bit surprised this issue seemingly hasn't come up for the GOF case. $\endgroup$
    – Meg
    Jan 27, 2015 at 15:12
  • 1
    $\begingroup$ I surprised too as this issue isn't discussed. I came up to the same solution as Glen but still have doubts. First, each pair is not independent from the "global" sample. For example, imagine we have 70,16,14 so you suggest compare 16 and 14 against 15/15. However, in other observation it could be 72,14,14. i.e. the source of "superiority' in the pair could be not a counterpart in the pair. Second, should we apply some group adjustment like Bonferroni if the choices weren't actually indepedent? Third, should we distinguish if the choice was mutually exclusive or it was multiple choice? $\endgroup$
    – Niksr
    Oct 31, 2015 at 2:03
  • $\begingroup$ I curious, could be possible to employ Cochran Q-test with McNemar post-hoc for this purpose? It seems all conditions for this test are met: 1) control stage - even distribution 2) event - reaction on stimuli 3) this is pair comparison (between hypothetical random choice and actual choice) 4) null - reaction on stimulus is different from random $\endgroup$
    – Niksr
    Oct 31, 2015 at 3:48
  • $\begingroup$ so you suggest compare 16 and 14 against 15/15 @Niksr, no. Glen compares the two groups as 50/50 percent. The 3rd group is excluded from the comparison. $\endgroup$
    – ttnphns
    Oct 31, 2015 at 12:14
  • $\begingroup$ Yes, I meant 16 and 14 are cases, not percents. $\endgroup$
    – Niksr
    Oct 31, 2015 at 17:55
4
$\begingroup$

I've had the same issue (and was happy to find this post). I now also found a short note on the issue in Sheskin (2003: 225) that I just wanted to share:

"Another type of comparison that can be conducted is to contrast just two of the original six cells with one another. Specifically, let us assume we want to compare Cell l/Monday with Cell 2/Tuesday [...] Note that in the above example, since we employ only two cells, the probability for each cell will be π_i = 1/2. The expected frequency of each cell is obtained by multiplying π_i = 1/2 by the total number of observations in the two cells (which equals 34). As noted previously, in conducting a comparison such as the one above, a critical issue the researcher must address is what value of alpha to employ in evaluating the null hypothesis."

Sheskin, D.J. 2003. Handbook of Parametric and Nonparametric Statistical Procedures: Third Edition. CRC Press.

Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.