1
$\begingroup$

I estimate a linear model consisting of one dependent and four independent variables. I want to test wether each of the independent variables has a significant impact on the dependent variable, i.e. I want to test four hypotheses. The dependent variable is some sort of "quality" indicating variable. As quality is a highly subjective term, researchers have brought forward 10 different ways of quantifying the quality variable I am interested in. I consider each one of these valid and thus want to repeat the estimation 10 times for each of the different dependent variables (that are all assumed to measure the same concept, i.e. some quality notion). If at least one of ten of the respective p-values indicates significance, I infer that there seems to be a relation between the respective coefficient and the quality notion I am interested in.

How do I adjust p-values in this case?

Do I have to adjust them twice? One time given the fact that I test four hypotheses with one model and one time given the fact that I repeat the test ten times? If the former is true, wouldn't I always have to adjust any p-value in any multiple regression model?

Improve this question
$\endgroup$
2
  • $\begingroup$ Could you please provide more information about what you mean when you say "repeat this test ten times with ten different variations of the dependent variable"? It's not clear that is a standard way to approach this type of problem. Please provide more information in the question (not in a comment, as comments can get lost) about the nature of your data and the goal of your study. it might be possible to reach the goal of your study without so much multiple comparison. $\endgroup$
    – EdM
    Commented Aug 10, 2020 at 14:51
  • $\begingroup$ I did. Please let me know if it is clear. $\endgroup$
    – shenflow
    Commented Aug 10, 2020 at 14:57

1 Answer 1

Reset to default
1
$\begingroup$

The one-measure-at-a-time approach is throwing away all the useful information from the correlations among the various measures of "quality," while leading to a massive multiple-comparison problem from the 10 separate regressions. And I'd be skeptical if you considered a particular predictor to be "significant" if it only was associated with a single quality measure.

This calls for a more direct multivariate (multiple-outcome) approach, in which the outcomes are evaluated together as functions of the predictors in a single model. See this document by Fox and Weisberg for an introduction. There are other multivariate approaches that take advantage of correlations both among predictors and among outcome measures, like partial least squares. Those multivariate approaches will get much more information out of your data while minimizing the multiple-comparison problem.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I understand. And what if I do not estimate one model ten times to test a set of hypothesis, but I estimate a set of models 10 times to test a set of hypothesis (e.g. one hypothesis per model). In that case, estimating a single multivariate model does not solve the issue I suppose, or does it? $\endgroup$
    – shenflow
    Commented Aug 10, 2020 at 17:26
  • $\begingroup$ @shenflow each separate hypothesis test needs to be taken into account with respect to multiple comparisons. The advantage of starting with one big model is that you start with a single overall estimate of whether there is any relationship at all between the predictors and the outcome(s). If so, then you can use established ways to handle multiple comparisons within a model that already is known to have some significance. Things like Bonferroni corrections start with the assumption that there might be no significant associations, but a significant overall test makes that already unlikely. $\endgroup$
    – EdM
    Commented Aug 10, 2020 at 17:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.