2
$\begingroup$

I have a dataset with four groups $G_i$ with $i = 1, 2 , 3 , 4$, and in each group $G_i$, a certain number $n_i$ of observations are favorable. I want to check if there are any differences between the groups. One way to do this is to compute the proportion of favorable outcomes in each group and to use pairwise $z$-tests. Is the chi-squared test appropriate here?

$\endgroup$

1 Answer 1

1
$\begingroup$

If your sole goal is to test in an overall way if the expected counts in each group differ from their observed counts, then the chi-square test is fine (specifically the chi-square test of uniformity). Note though that p-values from this test only give you the following: the probability of finding the observed chi-square given there are no differences between groups. It may be more helpful to first provide visualizations of the counts and to follow up with magnitudes of association such as the phi or Cramer $V$ coefficient. Both of these approaches only give you a general sense of the differences, and do not test where these shifts are coming from (aka specific groups). It is important to understand that there are some important assumptions to meet (probably one of the most important being independence between groups), so I suggest reading through this guide. For post-hoc comparisons, the paper by Franke et al. listed below explains the Goodman procedure for obtaining specific group differences.

If you don't want to approach it from a null hypothesis perspective (and avoid the p-value mental gymnastics), you can also consider a Bayesian approach, which I provide in the references for details on that.

References

  • Franke, T. M., Ho, T., & Christie, C. A. (2012). The chi-square test: Often used and more often misinterpreted. American Journal of Evaluation, 33(3), 448–458. https://doi.org/10.1177/1098214011426594
  • Jamil, T., Ly, A., Morey, R. D., Love, J., Marsman, M., & Wagenmakers, E.-J. (2017). Default “Gunel and Dickey” Bayes factors for contingency tables. Behavior Research Methods, 49(2), 638–652. https://doi.org/10.3758/s13428-016-0739-8
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.