I'm working on a tool to automatically analyze survey results.
The tool builds contingency tables and conducts statistical tests on them. When the questions have categorical (nominal) answer choices (e.g. "Eggs", "Waffles", "Pancakes"), I first conduct a chi-squared test of independence and then do a post hoc analysis to see which (if any) cells in the contingency table are contributing significantly to the result. The post-hoc test I'm using is the
standardized residuals test proposed by Agresti (2002) and discussed in Sharpe (2015).
I initially assumed that I would sometimes see table-level significance without cell-level but I did not expect the converse (i.e. cell-level significance without table level significance.) But I recently found an example dataset that produces that outcome.
If you run a chi-squared test on this table:
You get a p-value of 0.099 for the table overall. But if you proceed to post hoc analysis using standardized residuals, several of the cells several of the cells (highlighted) end up with p-values as low as 0.0016.
I suppose this might be a 'multiple comparisons' problem and that the significance is an illusion stemming from repeating the cell-wise test for each cell (35 times in this case). But even if I apply a Bonferroni correction (p-value * 35) then the "55 to 64"/"Do not plan to travel cell" would end up with a p-value of ~0.06. So if we did a test with a 93% significance level, we'd still end up with the table failing and the cell passing.
So...is it just conceptually invalid to do the post hoc analysis given the table p-value? (If it is, I'd appreciate an explanation of why). Or is it just natural and to be expected that cellwise dependence doesn't imply tablewise dependence?
I can provide the underlying data and code samples if you'd like to try this for yourself. But I'm using stat functions out of
scipy (with a little custom code for the standardized residuals part) so I'm reasonably confident that basic math isn't to blame.