I'm working on a tool to automatically analyze survey results.

The tool builds contingency tables and conducts statistical tests on them. When the questions have categorical (nominal) answer choices (e.g. "Eggs", "Waffles", "Pancakes"), I first conduct a chi-squared test of independence and then do a post hoc analysis to see which (if any) cells in the contingency table are contributing significantly to the result. The post-hoc test I'm using is the standardized residuals test proposed by Agresti (2002) and discussed in Sharpe (2015).

I initially assumed that I would sometimes see table-level significance without cell-level but I did not expect the converse (i.e. cell-level significance without table level significance.) But I recently found an example dataset that produces that outcome.

If you run a chi-squared test on this table:

Example crosstab

You get a p-value of 0.099 for the table overall. But if you proceed to post hoc analysis using standardized residuals, several of the cells several of the cells (highlighted) end up with p-values as low as 0.0016.

I suppose this might be a 'multiple comparisons' problem and that the significance is an illusion stemming from repeating the cell-wise test for each cell (35 times in this case). But even if I apply a Bonferroni correction (p-value * 35) then the "55 to 64"/"Do not plan to travel cell" would end up with a p-value of ~0.06. So if we did a test with a 93% significance level, we'd still end up with the table failing and the cell passing.

So...is it just conceptually invalid to do the post hoc analysis given the table p-value? (If it is, I'd appreciate an explanation of why). Or is it just natural and to be expected that cellwise dependence doesn't imply tablewise dependence?

I can provide the underlying data and code samples if you'd like to try this for yourself. But I'm using stat functions out of scipy (with a little custom code for the standardized residuals part) so I'm reasonably confident that basic math isn't to blame.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.