I found this article, describing a systematic approach to stratified analysis: https://www.linkedin.com/pulse/conducting-stratified-analysis-test-confounding-navneet-dhand
- Test crude association of the explanatory variable with the outcome or response variable, i.e. conduct a chi-square test to evaluate significance of the association and calculate a crude odds ratio or relative risk (along with their confidence intervals) to measures the strength of the association.
- Conduct stratified analysis after stratifying data by the third variable. Similar to the first step, this includes testing the significance and measuring the strength of the association, but for each contingency table created after stratification.
- Test homogeneity of odds ratios or relative risks to determine whether these measures of association are significantly different. This can be done by conducting a Breslow-Day or Woolf's test of homogeneity. A significant test indicates a significant interaction or effect modification. If this is the case, then it is preferable to report separate odds ratio and relative risk for each strata.
- Calculate Mantel-Haenszel odds ratios and relative risk along with their confidence interval, if the test of homogeneity is not significant. This is a weighted measure of association after adjusting for the third variable. The adjusted measure of association can be compared with the crude measure of association calculated in the first step to evaluate percentage change in the measure after adjusting for the third variable. A 'substantial' change is indicative of confounding. There are no set rules of deciding what change is substantial but generally more than 20% change is considered important.
- Conduct a Cochran- Mantel-Haenszel chi-square test to evaluate significance of the adjusted odds ratio or relative risk calculated in the fourth step.
This seems very useful to me. However, I also need to use multiple testing, e.g. the Benjamini-Hochberg procedure. There are a lot of different significance values described in every step of the process. Where should you use multiple testing correction, if this algorithm needs to be used on multiple samples?
We have a population with size $1,000,000$. We also have each person's sex and age. We have $25$ 2x2 contingency tables for some kind of variables we want to measure in the population. Obviously, if we conduct a test, e.g. the chi-squared test on all of these tables, we need to use multiple testing correction on the p-values. This is straightforward.
However, if we now want to stratify all of these tables by sex and age, and use the procedure described above, when and how should we use multiple testing? Is it correct to assume that multiple testing would fit best in the end of the algorithm for all of the chosen tables based on homogeneity in the stratified sets?