I am trying to understand whether the use of several linear models using the same variable as a response variable (but differing in one of the specified predictors) constitutes a case of multiple hypothesis testing.

My research is investigating the relationship between a continuous response variable (Y) and several continuous predictor variables (A, B, C, D). Additionally, I want to investigate if these relationships are dependent on two categorical variables (W, Z). Using a linear model, my ideal model specification would include 3-way interactions between each of my continuous predictors and the categorical predictors:

Y ~ A * W * Z + B * W * Z + C * W * Z + D * W * Z

Due to sample size limitations, I cannot fit all interactions of interest in the same model and instead have to specify each of them in separate models:

Model 1) Y ~ A * W * Z

Model 2) Y ~ B * W * Z

Model 3) Y ~ C * W * Z

Model 4) Y ~ D * W * Z

Does this constitute a case of multiple hypothesis testing for which corrections of significant p-values must be done? E.g., if the interaction between A and W (A * W) has a significant effect in Model 1), must its p-value be adjusted?

If so, which kind of correction is advisable?

  • 1
    $\begingroup$ Yes. Why can you not fit all interactions at once? $\endgroup$ – user2974951 Aug 2 at 10:18
  • $\begingroup$ Unfortunately I have a too small sample size to fit all those interactions. That's the reason I defined different models. $\endgroup$ – Rute Mendonça Aug 2 at 10:46
  • $\begingroup$ Then you are trying to fit too much. Building multiple models with subsets of terms is a dirty way of going around that. You need to reduce the number of terms in your one model. $\endgroup$ – user2974951 Aug 2 at 11:51
  • $\begingroup$ Not an answer but a piece of advice: before correcting for multiple hypothesis testing, check that your results of interest are significant. $\endgroup$ – Alexandre Cazenave-Lacroutz Aug 2 at 15:59
  • $\begingroup$ Thank you for your advice Alexandre. That's a very good point. What is the guideline there? That corrections for multiple hypothesis testing are to be considered when effects of interest are significant, but may be "dismissed" when those are not significant? $\endgroup$ – Rute Mendonça Aug 9 at 7:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.