7
$\begingroup$

This question comes from a discussion on the recent post by @rvl It's all in the family; but do we include the in-laws too?

Here's a common scenario that I've seen many times. A researcher runs a simple linear regression with, say, 5 covariates.

lm(Y ~ X1 + X2 + X3 + X4 + X5, data = df)

Ignoring interaction effects for the moment. They run the summary on the regression and observe the following:

      Estimate      Pr(>|t|)
X1    a            0.10
X2    b            0.47
X3    c            0.04            
X4    d            0.38
X5    e            0.12

From this, they conclude that covariate X3 is a significant predictor of outcome Y. I've seen this done many times.

My question is, why do we not have to adjust these $P$-values for multiple comparisons? Are we not doing 5 tests simultaneously, even though they are covariates, thus increasing the chances of seeing a false positive? Assuming 5 completely independent tests, there would be a $1-(1-\alpha)^M = 1-0.95^5 \sim 0.23$ or 23% chance of seeing a false positive, rather than than usual 5%, however this is in no way indicated in the reporting of the "significant association".

This article from the question Is adjusting p-values in a multiple regression for multiple comparisons a good idea? seems to indicate that if you are doing some kind of stepwise model selection, then it is advantageous to correct the $P$-values of your covariates to account for the increased type-1 error rate. This seems to indicate that tests covariates do not act differently than usual tests.

Has anyone had any experience with this? I would love to hear any flaws in my logic, or reasons why this should not be done.

$\endgroup$
  • 2
    $\begingroup$ Sorry I don't have time right now to give you a better answer, but the results from the regression are answering a particular hypothesis test (comparing distributions under the inclusion or removal of a particular variable. A Bonferroni-like adjustment would instead be more appropriate if the null hypothesis was more complicated including a vector of parameter. This is the case in multiple comparisons. $\endgroup$ – Jonathan Lisic Aug 16 '15 at 3:41
3
$\begingroup$

For the multiple testing problem it might be good to take a look at Family-wise error boundary: Does re-using data sets on different studies of independent questions lead to multiple testing problems?.

In your example above, if you estimate a regression on one sample, then you can, with a t-test only decide on the significance of an individual coefficient, so, yes, there is a multiple testing problem if you draw conclusions for multiple coefficients, based on multiple t-tests.

Let us call the coefficients $\beta_i, i = 1, 2, \dots 5$, then you can test $H_0^{(1)}: \beta_1 = 0$ versus $H_1^{(1)}: \beta_1 \ne 0$ with a t-test and conclude that $\beta_1$ is significant. Note that, if you can not reject $H_0^{(1)}$ that you can not conclude that $\beta_1$ is zero (see What follows if we fail to reject the null hypothesis?).

So if you want to find 'statistical evidence' for $\beta_1$ not being zero, then your $H_1^{(1)}$ must be the expression that you want to 'prove', i.e. $H_1^{(1)}: \beta_1 \ne 0$ and then $H_0^{(1)}$ is the opposite, i.e. $\beta_1=0$. As you assume $H_0^{(1)}$ to be true (to derive a statistical contradiction) you have a fixed value for the parameter $\beta_1=0$ and therefrom it follows that you know the distribution of the estimator $\hat{\beta}_1$ (see theory on linear regression) and you can compute p-values.

Let us now take the case where you want to show that $(\beta_1 \ne 0 \text{ and } \beta_2 \ne 0)$, then this must be your $H_1^{(1,2)}$ and the opposite $H_0^{(1,2)}$ is that either $(\beta_1 = 0 \text{ or } \beta_2 = 0)$, as there is an 'or' in there you can not fix all the parameters of the combined distribution of $(\hat{\beta}_1, \hat{\beta}_2)$ !

Can you apply multiple testing procedures ? Most of them assume that the individual p-values are independent, in this example $\hat{\beta}_1$ and $\hat{\beta}_2$ can not be shown to be independent !

But, in an advanced book on econometrics (e.g. W.H. Greene, "Econometric Analysis") you will find applicable test for J (simultaneous) linear restrictions ($\beta_i=0, i=1,2,3,4,5$ is a special type of 5 linear restrictions) that avoid the multiple testing problem.

$\endgroup$
  • 4
    $\begingroup$ (+1) Note the usual omnibus F-tests for simultaneous linear restrictions indeed take account of multiple comparisons; but they don't provide for rejection of individual constituents of the composite null. $\endgroup$ – Scortchi - Reinstate Monica Aug 17 '15 at 9:13
  • 1
    $\begingroup$ @fcoppens, Sorry for the delay in my response. I enjoyed reading your answer, thank you. It's trivial that $\hat{\beta}_1$ and $\hat{\beta}_2$ are not independent, but I don't think it is correct that all multiple testing procedures require independent tests. Notably, Bonferroni ( and some derivatives) does not require independence and FDR holds under positive regression dependancy. Just wanted to let you know. Otherwise very informative answer, and I"ll look into W.H. Greene. Cheers. $\endgroup$ – Chris C Aug 17 '15 at 14:26
  • $\begingroup$ @Scortchi, Thanks for the reminder. However, model selection for variable inclusion in an omnibus test based on $P$-values is subject to this problem, so they aren't immune to the problem. Not to say model selection is always done, but it is occasionally, especially in the case of interaction effects. You're right, just my 0.02. $\endgroup$ – Chris C Aug 17 '15 at 14:31
  • $\begingroup$ @ChrisC: you are right (+1) on bonferonni, it does not assume independence, but it is conservative. For FDR I must check but for 5 hypothesises FDR is not so relevant and control of FDR does not imply control of FWER. $\endgroup$ – user83346 Aug 17 '15 at 14:53
  • 1
    $\begingroup$ @ChrisC: I just wanted to point out that the F tests referenced don't correct p-values for tests that individual coefficients are equal to zero to control the regression-wise error rate - I don't know if there's any practical higher-powered alternative to the Bonferroni method for that. Not quite sure what you mean with "model selection for variable inclusion in an omnibus test based on P-values", but if you want to use hypothesis testing for model selection, carrying out a small no. of chunk tests ameliorates the multiple-testing issues. $\endgroup$ – Scortchi - Reinstate Monica Aug 18 '15 at 9:55
6
$\begingroup$

There may be a few additional aspects worth considering (which are a little too long for a comment).

  1. Whether or not there is a multiple testing problem in a given application quite strongly depends on which coefficients a researcher looks at. In many applications, one is only interested in 1-2 key variables, and the others only act as "controls". Say, in a fixed effects panel data model we may feel that we need individual specific intercepts to control for unobserved heterogeneity, but we are typically not really interested in these $N$ fixed effects per se. On the other hand, in for example growth econometrics, we sift through all possible determinants for growth and as such, we are willing to look at all significant variables. In the latter case, we do have a multiple testing problem, but not necessarily in the former.

  2. I would argue that there are indeed several high-powered (at least, higher powered than Bonferroni) alternatives for performing such a model selection exercise. These include Bayesian model averaging, extreme-bounds analysis, General-to-specific, penalized methods (Lasso and related methods) and also methods directly deriving from the multiple testing literature. The latter group includes classical ones based on the Benjamini-Hochberg method, but also more recent bootstrap-based methods. To do some shameless self-promotion, these are compared and applied in a paper of mine.

$\endgroup$
  • $\begingroup$ (+1) Thank you, this is a fantastic post. I had not considered point 1, and you are right that it makes quite the difference. I'm afraid that this makes the whole thing subjective, as a priori tends to slur to when I have time occasionally. Would you recommend this dichotomy between "control" and "interest" variables? I have not had a chance to read your paper as of yet, but I have it book marked and will give it a read as soon as I have the chance, thank you for the link. Best. $\endgroup$ – Chris C Aug 20 '15 at 13:11
  • $\begingroup$ Hm, it is hard to say in general whether that dichotomy is recommendable - will likely depend on the application. And although I presented it as a black and white thing, I of course see that the distinction may be quite blurred in practice. $\endgroup$ – Christoph Hanck Aug 20 '15 at 13:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.