4
$\begingroup$

Possible Duplicate:
Not-significant F but a significant coefficient in multiple linear regression
How can a regression be significant yet all predictors be non-significant?
Significance of coefficients in linear regression: significant t-test vs non-significant F-statistic

If in a multiple linear regression (enter method) the general model isn't significant (F>.05) but one of the predictors is significant (β<.05), should I consider it as a significant result?

$\endgroup$

marked as duplicate by Macro, gung, whuber Aug 21 '12 at 19:26

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • $\begingroup$ Did that actually happen? I wouldn't expect it to. $\endgroup$ – Michael Chernick Aug 21 '12 at 17:35
  • $\begingroup$ I voted to close as a duplicate. I didn't take the time to find the exact thread but questions along these lines have been asked on here many times and I'd suggest doing a search of the site to find your answer. The basic point is that you're doing two different hypothesis tests so you can't expect them to agree all of the time. $\endgroup$ – Macro Aug 21 '12 at 18:54
  • 2
    $\begingroup$ @Michael This is not at all hard to reproduce, knowing that including lots of irrelevant variables will increase the overall p-value. Here's a case with up to five variables where you can watch the overall p-value balloon as each variable is added, while the individual p-value changes little. Try this in R: set.seed(17); p <- 5; x <- as.matrix(do.call(expand.grid, lapply(as.list(1:p), function(i) c(-1,1)))); y <- x[,1] + rnorm(2^p, sd=2); temp <- sapply(1:p, function(i) print(summary(lm(y ~ x[, 1:i])))). $\endgroup$ – whuber Aug 21 '12 at 19:55
  • $\begingroup$ @whuber Yes I realized that when I read the answer provided by Peter and gung. $\endgroup$ – Michael Chernick Aug 21 '12 at 19:58
3
$\begingroup$

I think @Macro is right; in fact, I'm pretty sure I've answered exactly this question. However, I can't find it, and I think the underlying reason for this happening is unrelated to the reason for the reverse situation. So I'll put down some quick information.

One thing that I think is unfortunate is that the problem of multiple comparisons is always discussed the same / in only one way, namely the comparisons between multiple groups. But this issue occurs everywhere, not just in that situation. For example, if you run a multiple regression with 20 covariates where the null hypothesis obtains for each of them, you should expect that in the long run, on average one of them will appear 'significant' anyway in each model. There are various ways of addressing this issue (e.g., alpha correction techniques), but the most common is to use a simultaneous test, that is, a global $F$ test of the model.

Thus, my first guess is that the model test is doing its job and protecting you from inflated family-wise type I error. That is, it is telling you to ignore any possible significant betas. (I apologize if this is bad news.)

For the sake of completeness, there is of course, another possibility. The power of a simultaneous test can be quite week, especially in cases where there are many unrelated covariates, only a few (or one) actually related covariates that are weakly correlated with the response, and a large error term. Despite that fact, you should be cautious about concluding that the beta in question is 'significant'.

$\endgroup$
  • $\begingroup$ With all due respect, @gung, if you're pretty sure you've answered this exact question then it seems like perhaps you should be casting a close vote. High rep members like yourself are relied upon to lead by example. I'm not sure answering questions you know are duplicates is a wonderful example. $\endgroup$ – Macro Aug 21 '12 at 19:01
  • 5
    $\begingroup$ That's a valid point, @Macro, but if I (as a high rep member w/ some sense of where & how to look) can't find it after 5 minutes of searching, what chance does someone new to the site have? Instead of spending more of my time trying to figure out how to find it, I spent 5 minutes answering this Q. I recognize there are cons to this approach, but there are pros as well. I imagine it's more helpful for the OP, & it will be easier for future visitors to find a needle in a haystack if there are more needles. I just don't think the cons outweigh the pros in this case. $\endgroup$ – gung Aug 21 '12 at 19:06
  • $\begingroup$ Well, after looking for a minute or two I found one close match here. There are probably more somewhere. I'd still say the OP has a responsibility to search the site before asking a question and this kind of attitude on the part of experienced users discourages that. Answers are submitted for future users as well as the OP - answering questions as they come up without looking for the duplicates seems to reflect an attitude that answers are for the OP, not the community. $\endgroup$ – Macro Aug 21 '12 at 19:11
  • 1
    $\begingroup$ @gung I sympathize; it's pretty hard sometimes to find a duplicate you know is there. I managed to find a close one just by scanning down the "related" links automatically offered by the system (see at the right). When one of those is a near hit, you can be pretty sure that either the OP did little or no research while writing the question or they are not sufficiently conversant with the terminology to recognize a duplicate. It's good to take some care in the latter case to explain why something is a duplicate--but then close the question anyway. $\endgroup$ – whuber Aug 21 '12 at 19:29
  • 2
    $\begingroup$ @Michael One is torn between competing objectives: closing quickly can help the OP by directing them immediately to a solution. Leaving a question open allows it to collect answers which then later will either disappear or will have to be merged with the duplicate, which can create a little confusion (as well as quite a bit more work for the mods :-). Because a closed question is still visible, can still be discussed (as we are doing here), and can easily be reopened, I find it's best to err on the side of closing so that such early answers do not accumulate. $\endgroup$ – whuber Aug 21 '12 at 20:10
2
$\begingroup$

This happens because the general model and the specific parameters answer different questions. Whether you should consider the result significant depends on what your original hypotheses are. If they were about the significant variable, controlling for the others, then I'd say "yes". If they were about the whole model, I'd say "No".

In either case, I'd stress effect sizes rather than significance levels.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.