2
$\begingroup$

I'm working on a tool to automatically analyze survey results.

The tool builds contingency tables and conducts statistical tests on them. When the questions have categorical (nominal) answer choices (e.g. "Eggs", "Waffles", "Pancakes"), I first conduct a chi-squared test of independence and then do a post hoc analysis to see which (if any) cells in the contingency table are contributing significantly to the result. The post-hoc test I'm using is the standardized residuals test proposed by Agresti (2002) and discussed in Sharpe (2015).

I initially assumed that I would sometimes see table-level significance without cell-level but I did not expect the converse (i.e. cell-level significance without table level significance.) But I recently found an example dataset that produces that outcome.

If you run a chi-squared test on this table:

Example crosstab

You get a p-value of 0.099 for the table overall. But if you proceed to post hoc analysis using standardized residuals, several of the cells several of the cells (highlighted) end up with p-values as low as 0.0016.

I suppose this might be a 'multiple comparisons' problem and that the significance is an illusion stemming from repeating the cell-wise test for each cell (35 times in this case). But even if I apply a Bonferroni correction (p-value * 35) then the "55 to 64"/"Do not plan to travel cell" would end up with a p-value of ~0.06. So if we did a test with a 93% significance level, we'd still end up with the table failing and the cell passing.

So...is it just conceptually invalid to do the post hoc analysis given the table p-value? (If it is, I'd appreciate an explanation of why). Or is it just natural and to be expected that cellwise dependence doesn't imply tablewise dependence?

I can provide the underlying data and code samples if you'd like to try this for yourself. But I'm using stat functions out of scipy (with a little custom code for the standardized residuals part) so I'm reasonably confident that basic math isn't to blame.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.