How to find out which levels of a categorical variable are different comparing 2 groups?

Question

I performed a chi-square test for 2 groups on a categorical variable. The p-value indicates that the categorical variable is significant.

I concluded that at least one level of the categorical variables differ between 2 groups.

How can I find which levels are actually different and which levels are not different between 2 groups? For instance if a categorical variable has 3 levels, married, divorced and single, which test should I conduct to be able to say, married people significance have changed but divorced people are not significantly different between 2 groups?

I am conducting the tests in python and I used chi2_contingency from scipy.stats.

Answer 1

A common approach is to look at the standardized residuals (a.k.a. Pearson residuals), or to the adjusted standardized residuals. See Donald Sharpe's paper "Chi-Square Test is Statistically Significant: Now What?" (2015), that is a short review of residual analysis and other methods to use after a chi-squared test. Page 3 in particular is useful for the interpretation of adjusted residuals and how to correct for multiple testing.

As you mention working with Python, both types of residuals are available with the statsmodels library in Python, through the methods resid_pearson (Pearson residuals) and standardized_resids (adjusted standardized residuals). See the statsmodels documentation on contingency tables to get an idea of how to use it.

Note that conducting residual analysis conditioned on the result of a chi-squared test has been criticized, because this approach loses control of type 1 error rates; see García-Pérez et al. (2015). The authors offer a solution to this issue, implemented in R and Matlab (but not Python I'm afraid). However, their method is apparently not used a lot, and for this reason may lack some scrutinity – which is unfortunate if you were looking for a tried and tested method; in this case you may simply want to stick to the methods reviewed by D. Sharpe.

---

References

Sharpe, Donald (2015) "Chi-Square Test is Statistically Significant: Now What?", Practical Assessment, Research, and Evaluation, Vol. 20, Article 8. DOI: https://doi.org/10.7275/tbfa-x148

García-Pérez, M.A., Núñez-Antón, V. & Alcalá-Quintana, R. "Analysis of residuals in contingency tables: Another nail in the coffin of conditional approaches to significance testing." Behav Res, 47, 147–161 (2015). https://doi.org/10.3758/s13428-014-0472-0