4
$\begingroup$

This may be a really basic question, but I haven't found any helpful answers. Or I simply don't understand.

When correcting p-values of linear regression models, which p-values are corrected?

My problem:

I ran two multiple linear regression models with each 3 predictors and one dependent variable. Now I would want to run a FDR (Benjamini-Hochberg) correction. But I am not sure which p-values to correct and how.

1) I thought about correcting the p-values of all 6 coefficients of both models (t-statistics). I would sort them in ascending order, following this approach.

Would this be the correct way for adjusting the p-values? And what about the p- values of the F-Statistic of both models?

Thank you in advance.

$\endgroup$
0
5
$\begingroup$

It is good that you are concerned with the type I error rate of running multiple models. However, keep in mind that any kind of correction will come at the cost of an increased type II error rate: By being more strict you will lose power.

Hence, only perform and correct for tests that are directly relevant to your research question. A summary of a linear model may automatically generate a $p$-value for the intercept, but if this test is meaningless (as it usually is), correcting for it would be wasteful in terms of power, if it weren't significant. If it is significant and you use e.g. the Benjamini Hochberg procedure, you could even inflate your false positive rate by including a nonsense result.

Another example is including confounders in your model. If you include a variable because you have reason to assume there might otherwise be omitted-variable bias, then its significance is not relevant. It need neither be checked, nor corrected for.

Likewise, suppose your research question revolves around the presence (or absence) of an interaction. In this case, significance of the marginals is irrelevant.

To conclude, first determine which $p$-values are actually of interest, then correct for those.

$\endgroup$
2
  • $\begingroup$ Thank you very much for your answer! Could you may be explain your last point, since I am not sure I understood. I do have the research questions based on the presence/absence of an interaction. However, the interactions are not significant but a predictor alone is. I now wonder since I ran multiple models, the p-values of all coefficients (not only the Interaction term) should be corrected? $\endgroup$ – Guest67 Jul 13 '19 at 7:49
  • 1
    $\begingroup$ @Guest67 you seem to be hung up on statistical significance. This is problematic. Think more about pre-specification of models using subject matter expertise, without removing "insignificant" terms. $\endgroup$ – Frank Harrell Jul 13 '19 at 11:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.