6
$\begingroup$

Okay, this is a thought experiment:

Suppose you have a dataset with 40 covariates. Suppose the data has every nice theoretical property you could want: randomly sampled, variables not correlated, etc. Suppose it is also the case that the null is true for each beta coefficient at a 95% level AND we don't know that the null is true in all cases.

Next, you run the model, and 2 of the 40 betas appears to be significant, with a modestly large t value, but nothing crazy. Certainly enough to reject the null for both of these betas. Now, given that it was assumed the null is true, this is a perfectly reasonable result. Each beta has an approximately 5% chance of rejecting, so you would have expected two betas to reject before you even run the model!

This brings me to my issue: in what sense can we say multiple betas are significant when we ALWAYS expect a certain number to reject just due to randomness? AND how is it possible to conduct hypothesis tests for these betas simultaneously? Isn't that cheating, sort of like rolling a die until you get a six? It seems to me you can only pre-specify a single beta to look at before you run the model to get "true" significance.

To clarify what I mean by simultaneous: I am not talking about the overall test (F-Test, for example) or any related mitigating actions like multiple comparison corrections. I fully agree that these tests and measures are effective and make sense. I am asking at the level of the individual betas themselves with their individual CIs: if two betas reject, should I just ignore this and say: "Well, I expected two to reject. These effects are not real"? This may be my main point: does significance at the regression coefficient level mean anything in this case? Or suppose four had rejected. I would guess two would be false signals by chance, but how do I know which is real and which is fake?

PS: If you have an open-source text to cite and support your answer, I would love that.

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ The standard procedure for simultaneous testing is based on the F-ratio statistic. $\endgroup$
    – whuber ♦
    Commented Jan 31, 2020 at 18:05
  • $\begingroup$ I mentioned above that I'm talking about the betas individually, and not the model as a whole. The F test will only tell us if at least one beta is useful, which is different from my question. I'm asking how we can know if each beta is simultaneously statistically significant, not if at least one is statistically significant considering the betas simultaneously. $\endgroup$
    – user271536
    Commented Jan 31, 2020 at 18:16
  • 1
    $\begingroup$ I think you mischaracterize the situation: the F test will tell whether a group of betas is simultaneously significant. $\endgroup$
    – whuber ♦
    Commented Jan 31, 2020 at 18:24
  • 1
    $\begingroup$ Yes, I don't think I've been clear. I've tried to clarify above. $\endgroup$
    – user271536
    Commented Jan 31, 2020 at 18:54

2 Answers 2

Reset to default
3
$\begingroup$

It's true that with 40 variables, you would expect two to be significant (when using the conventional alpha of .05) by chance alone. That's why we shouldn't interpret the individual t-tests for a multiple regression model until after assessing the F-test for the model taken as a whole. (It may help you to read my answer here: Significance contradiction in linear regression: significant t-test for a coefficient vs non-significant overall F-statistic.) If I had a model with a non-significant F-test, but 2 out of 40 variables were individually significant, I would interpret those results as not actually meaningful.

To be clear, I would not quite say, 'these effects are not real', because that isn't a valid interpretation of a non-significant result (see: Why do statisticians say a non-significant result means “you can't reject the null” as opposed to accepting the null hypothesis?) I would say that you don't have sufficient evidence to reject the null.

Regarding the follow-up question (how to know which 2 of 4 are real and which fake), you'll never know if significant variables have a "real" relationship with the response, and you'll never know if non-significant variables don't. That's part of the nature of the game we're playing. If you were really concerned about the possibility that some of your results might be false discoveries, you could use false discovery procedures to explore that, but it isn't common to do so in the kind of situation you describe.

Improve this answer
$\endgroup$
6
  • $\begingroup$ So here's the trick. With the 2 false, 2 true case, we can note that 50% of the signals are false with probability. And this % would change to 33% if there are 2 false and 4 true. But if we prespecify a single regression coefficient of interest, and then ignore the coefficients for all other variables, we have an unchanging 5% chance of committing a type I error. So it seems ill advised to ever look at a subset of regression coefficients since the probability of a signal being true jumps around depending on how many true signals you have. But much is lost by looking at one beta! Thoughts? $\endgroup$
    – user271536
    Commented Feb 1, 2020 at 3:31
  • $\begingroup$ So basically, the chances of being wrong jumps around depending on how many true signals and false signals there are. That just seems unscientific and requires that I find the expected number of false signals for a given regression (no big deal, but clunky). But if we look at a single beta, it's the same percent change of having Type I error every time. But what a waste of other betas... $\endgroup$
    – user271536
    Commented Feb 1, 2020 at 3:34
  • $\begingroup$ @Trillium, to start with, it's rather poor science to fit a model w/ 40 variables just to see what might be significant. That's just a fishing trip. Usually, there are at most a few focal (intervention / exposure) variables of interest, & the rest are just control variables. When there's only 1 variable whose test you are going to interpret, you're back to .05 b/c it's a-priori. $\endgroup$
    – gung - Reinstate Monica
    Commented Feb 1, 2020 at 4:17
  • $\begingroup$ I have worked on a couple studies w/ large numbers of variables, & what I've done is group the variables into sets (eg, demographics, a couple categories of diagnoses, the existence of certain types of prescriptions, measures of how sick the patients are, etc), then, those groups are tested together w/ nested model tests. Then, >40 variables becomes ~8 tests. It's still a lot, but it's not as bad. Again, if you really have a lot of variables & it's a complete fishing trip, then you can use false discovery procedures, but those are mostly common in GWAS studies. $\endgroup$
    – gung - Reinstate Monica
    Commented Feb 1, 2020 at 4:20
  • $\begingroup$ This is helpful, thank you. I upvoted your answer, and think it's definitely the most useful response I've gotten, but I don't feel the topic is resolved for me. Partly because I'm having trouble expressing it. It just seems like the certainty we have about the signal of a coefficient depends a bit on how many other betas there are, and I feel like it should not--the assessment of the signal should depend on nothing else. I feel insecure on the behalf of the subject of statistics itself because of this! But maybe it just does depend, that is the answer, and I should stop worrying. $\endgroup$
    – user271536
    Commented Feb 1, 2020 at 4:33
2
$\begingroup$

If I'm reading your question correctly you are simply asking about multiple comparison in statistics which is a well known phenomena. To remedy this, you can correct your significance using something like a Bonferroni-correction, although many types of correction methods exist.

Note that you also need to consider the implication of your analysis. Is it something that will dictate how your company or medical department operates? Then you should probably take it more seriously than if your study is simply to pave way and direct future research. Multiple comparison and how we address it, is in essence just a dance between type-1 and type-2 errors.

EDIT: After reading your edit.

This may be my main point: does significance at the regression coefficient level mean anything in this case?

The coefficient tells you both the direction and size of the impact your covariates have on the dependent variable, with respect to the model as a whole. If the p-value of a covariate is significant, but the model contains enough individual covariates with no pre-determined hypothesis, that you suspect the unadjusted p-values may be erroneous, the coefficient and the CI may be inflated, even if truly significant. Another important part with respect to coefficients is how well you have controlled for confounders as these can dramatically change the coefficients in your model. Related literature on this topic.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks Paze, I've tried to clarify. I do feel my question is different from multiple comparison. See if you agree in light of my updates. $\endgroup$
    – user271536
    Commented Jan 31, 2020 at 18:57
  • $\begingroup$ I've edited my answer in light of this. You are in essence asking if the betas and CI's of a variable are "useless" if the p-value is significant at alpha = 0.05, but the model contains enough variables to produce such an effect by chance. The answer is difficult but boils down to "not necessarily". Again it depends on the impact of your research, as I wrote above. I have published papers with a lot of variables and surely none would be significant at a corrected alpha, but the aim of the research was simply to shed light on unknown topics. $\endgroup$
    – Paze
    Commented Jan 31, 2020 at 19:10
  • $\begingroup$ I think "not necessarily" is accurate, but this allows for "but possibly", which is this case in my contrived example: we know the betas are useless. This is sad to me, because if I am understanding this correctly, the power of regression is much less than people think for understanding predictors. So it seems to me that when we model, we can often say little about the relationship between individual covariates and the response, but only that the covariates are important (and it is not clear which are important). Of course, if a p-value is astonishingly small, then I think we can say more. No? $\endgroup$
    – user271536
    Commented Jan 31, 2020 at 19:17
  • $\begingroup$ Again I urge to consider this in light of what your research question is. What are the implications of thinking you are right, but you are actually wrong? What are the implications of thinking you are wrong, but you are actually right? If my research question is: "What factors may be associated to hip fractures in the elderly?" I could run 40 hypotheses and report them at alpha = 0.05 as long as I am truthful about my p-values being unadjusted and that the study is an exploratory study for future research. $\endgroup$
    – Paze
    Commented Jan 31, 2020 at 19:23
  • $\begingroup$ I've also had situations where I could not attain adjusted significance with individual variables but I could clearly see a pattern and there was truly something there, I just couldn't "prove" it. Turns out the pattern was real. Sometimes simply looking at your data may also give you more insight than your model and one should always start there. Visualizing data with expert knowledge (of the field of research, not necessarily statistics) is often very powerful. $\endgroup$
    – Paze
    Commented Jan 31, 2020 at 19:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.