Is it invalid to conduct post-hoc analysis on a contingency table when an initial chi-squared test shows no significant evidence against independence? I'm working on a tool to automatically analyze survey results.
The tool builds contingency tables and conducts statistical tests on them. When the questions have categorical (nominal) answer choices (e.g. "Eggs", "Waffles", "Pancakes"), I first conduct a chi-squared test of independence and then do a post hoc analysis to see which (if any) cells in the contingency table are contributing significantly to the result. The post-hoc test I'm using is the standardized residuals test proposed by Agresti (2002) and discussed in Sharpe (2015).
I initially assumed that I would sometimes see table-level significance without cell-level but I did not expect the converse (i.e. cell-level significance without table level significance.) But I recently found an example dataset that produces that outcome.
If you run a chi-squared test on this table:

You get a p-value of 0.099 for the table overall. But if you proceed to post hoc analysis using standardized residuals, several of the cells several of the cells (highlighted) end up with p-values as low as 0.0016. 
I suppose this might be a 'multiple comparisons' problem and that the significance is an illusion stemming from repeating the cell-wise test for each cell (35 times in this case). But even if I apply a Bonferroni correction (p-value * 35) then the "55 to 64"/"Do not plan to travel cell" would end up with a p-value of ~0.06. So if we did a test with a 93% significance level, we'd still end up with the table failing and the cell passing.
So...is it just conceptually invalid to do the post hoc analysis given the table p-value? (If it is, I'd appreciate an explanation of why). Or is it just natural and to be expected that cellwise dependence doesn't imply tablewise dependence?
I can provide the underlying data and code samples if you'd like to try this for yourself. But I'm using stat functions out of scipy (with a little custom code for the standardized residuals part) so I'm reasonably confident that basic math isn't to blame.
 A: There's a paper from 2014 about this issue: García-Pérez, M.A., Núñez-Antón, V. & Alcalá-Quintana, R. Analysis of residuals in contingency tables: Another nail in the coffin of conditional approaches to significance testing. Behav Res 47, 147–161 (2015). https://doi.org/10.3758/s13428-014-0472-0.
Unfortunately the paper is still behind a paywall as of 2022, but in short they show that testing residuals only after a significant chi-square test of independence loses control of type I error rates. Your question is good, but what I retain from their paper is that the procedure you describe is inadequate even when the test is significant.
So you should avoid your standard procedure altogether (i.e. looking at the chi-square test and then looking at the standardized residuals), and consider other approaches.
As alternatives, García-Pérez et al. mention two possibilities:

*

*Outlier detection, i.e. ignoring the omnibus test and focusing on analyzing deviant cells (which is a bit akin to what you suggest, but use different methods to analyze the cells). García-Pérez et al. don't study this kind of method in their paper, as they consider that an omnibus test may still be relevant (depending on the analysis purpose). However, their introduction mentions various references about detecting outliers, e.g. Outlier Detection in Contingency Tables based on Minimal Patterns by Sonja Kuhnt et al. (2014). For implementations, you can find several R packages that deal with detecting outliers in contingency tables, e.g. https://search.r-project.org/CRAN/refmans/alphaOutlier/html/aout.conttab.html for an implementation of the Kuhnt et al.'s paper, or https://github.com/mlindsk/molic for a more recent method (2019). I'm not aware of implementations in other languages, but I didn't look extensively into it. It may be a low-hanging fruit for some Python developers out there.

*A one-stage procedure involving bootstrapping, that García-Pérez et al. detail in their paper (I quote):



*

*compute the marginal probability distributions in the empirical table
under analysis;


*generate T tables of the same size and with the same overall number of observations under the model of concern (i.e., independence or
homogeneity) using those marginal probability distributions;


*find out numerically the value α* such that 100α% of the T tables have one or more significant residuals at α*;


*for the empirical table under analysis, reject:
a. the residual hypothesis for every cell whose residual is significant at α, and
b. the omnibus hypothesis if at least one of the residuals is significant
at α*.

Fortunately, they offer freely accessible code (in MATLAB and R) to reproduce this procedure. The ACT.r file is where you'll find the core of the procedure, the other files are very simple examples of how to use it.
Note that their paper does not really recommend applying this approach to multiway contingency tables, as they only studied the two-way case.
