I found this article, describing a systematic approach to stratified analysis: https://www.linkedin.com/pulse/conducting-stratified-analysis-test-confounding-navneet-dhand

  1. Test crude association of the explanatory variable with the outcome or response variable, i.e. conduct a chi-square test to evaluate significance of the association and calculate a crude odds ratio or relative risk (along with their confidence intervals) to measures the strength of the association.
  2. Conduct stratified analysis after stratifying data by the third variable. Similar to the first step, this includes testing the significance and measuring the strength of the association, but for each contingency table created after stratification.
  3. Test homogeneity of odds ratios or relative risks to determine whether these measures of association are significantly different. This can be done by conducting a Breslow-Day or Woolf's test of homogeneity. A significant test indicates a significant interaction or effect modification. If this is the case, then it is preferable to report separate odds ratio and relative risk for each strata.
  4. Calculate Mantel-Haenszel odds ratios and relative risk along with their confidence interval, if the test of homogeneity is not significant. This is a weighted measure of association after adjusting for the third variable. The adjusted measure of association can be compared with the crude measure of association calculated in the first step to evaluate percentage change in the measure after adjusting for the third variable. A 'substantial' change is indicative of confounding. There are no set rules of deciding what change is substantial but generally more than 20% change is considered important.
  5. Conduct a Cochran- Mantel-Haenszel chi-square test to evaluate significance of the adjusted odds ratio or relative risk calculated in the fourth step.

This seems very useful to me. However, I also need to use multiple testing, e.g. the Benjamini-Hochberg procedure. There are a lot of different significance values described in every step of the process. Where should you use multiple testing correction, if this algorithm needs to be used on multiple samples?


We have a population with size $1,000,000$. We also have each person's sex and age. We have $25$ 2x2 contingency tables for some kind of variables we want to measure in the population. Obviously, if we conduct a test, e.g. the chi-squared test on all of these tables, we need to use multiple testing correction on the p-values. This is straightforward.

However, if we now want to stratify all of these tables by sex and age, and use the procedure described above, when and how should we use multiple testing? Is it correct to assume that multiple testing would fit best in the end of the algorithm for all of the chosen tables based on homogeneity in the stratified sets?


1 Answer 1


This type of naive procedure will not ever keep the type 1 error rate or FDR (or would require extreme contortions to do so). In fact, it is also a pretty poor approach to what is effectively model building with some steps using unadjusted analyses (why??) or several steps being conditional on not rejecting a null. That kind of thing is simply guaranteed to be problematic. Combining a more reasonable exploratory model building approach that has some chance of producing reasonably calibrated SEs/CIs with e.g. Hochberg (or using a Bayesian hierarchical shrnkage model from the start) might be a better way of approximately controlling the FDR.

  • $\begingroup$ I am using the Benjamini-Hochberg false discovery rate for multiple testing. I do not, however, understand your comment on why the stratified analysis procedure described above is a poor approach in model building. Could you elaborate a bit? What would be the more reasonable approach? $\endgroup$
    – kiitr
    Mar 30, 2017 at 9:12
  • $\begingroup$ Also, is the Bayesian hierarchical shrinkage model explicitly better in every case? $\endgroup$
    – kiitr
    Mar 30, 2017 at 9:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.