91
$\begingroup$

In a multiple linear regression, why is it possible to have a highly significant F statistic (p<.001) but have very high p-values on all the regressor's t tests?

In my model, there are 10 regressors. One has a p-value of 0.1 and the rest are above 0.9


For dealing with this problem see the follow-up question.

$\endgroup$
7
  • 2
    $\begingroup$ Is the constant also insignificant? How many cases are involved? How many variables? $\endgroup$
    – whuber
    Commented Aug 19, 2011 at 5:34
  • $\begingroup$ How was multicollinearity diagnosed? There are many methods, some are more informative than others. The more you tell us, the better the community can answer. $\endgroup$
    – StasK
    Commented Aug 19, 2011 at 14:43
  • 5
    $\begingroup$ This question has become a FAQ. Some of the answers here were merged from substantially similar threads. $\endgroup$
    – whuber
    Commented Aug 7, 2012 at 15:50
  • 3
    $\begingroup$ See also here: how can a regression be significant yet all predictors be non-significant, & for a discussion of the opposite case, see here: significant t-test vs non-significant F-statistic. $\endgroup$ Commented Sep 13, 2012 at 15:15
  • 1
    $\begingroup$ Although all three answers say "multicollinearity," that's actually a special circumstance. The point of using an F test in the first place is that individual p-values for a group of regressors can give conflicting information about the significance of the group as a whole. See the related threads that @gung links to. $\endgroup$
    – whuber
    Commented May 11, 2016 at 20:23

10 Answers 10

65
$\begingroup$

As Rob mentions, this occurs when you have highly correlated variables. The standard example I use is predicting weight from shoe size. You can predict weight equally well with the right or left shoe size. But together it doesn't work out.

Brief simulation example

RSS = 3:10 #Right shoe size
LSS = rnorm(RSS, RSS, 0.1) #Left shoe size - similar to RSS
cor(LSS, RSS) #correlation ~ 0.99

weights = 120 + rnorm(RSS, 10*RSS, 10)

##Fit a joint model
m = lm(weights ~ LSS + RSS)

##F-value is very small, but neither LSS or RSS are significant
summary(m)

##Fitting RSS or LSS separately gives a significant result. 
summary(lm(weights ~ LSS))
$\endgroup$
3
  • 14
    $\begingroup$ It is interesting and important to note that both of your models predict equally well, in this case. High correlations among predictors are not necessarily a problem for prediction. Multicolinearity is only a problem when 1) analysts try to inappropriately interpret multiple regression coefficients; 2) the model is not estimable; and 3) SEs are inflated and coefficients are unstable. $\endgroup$
    – Brett
    Commented Jun 9, 2011 at 14:27
  • 1
    $\begingroup$ I understand that the two variables are highly correlated with each other, thus the result of t test is non-significant while the result of F test is significant. But how come this happens? I mean, what is the reason underlies this fact? $\endgroup$
    – yue86231
    Commented Jun 15, 2014 at 17:57
  • $\begingroup$ @yue86231 The t-test tests whether a predictor predicts any variability in the outcome given that all other predictors are in the model. The F-test tests whether the predictors jointly explain any variability in the outcome. Given the left shoe size, the right shoe size explains little variability in the outcome. Similarly, given the right shoe size, the left shoe size explains little variability in the outcome. But jointly, left and right shoe size explain a lot of variability in the outcome. So the t-tests are nonsignificant but the F-test is. $\endgroup$
    – Noah
    Commented Sep 20, 2023 at 4:50
126
$\begingroup$

It takes very little correlation among the independent variables to cause this.

To see why, try the following:

  • Draw 50 sets of ten vectors $(x_1, x_2, \ldots, x_{10})$ with coefficients iid standard normal.

  • Compute $y_i = (x_i + x_{i+1})/\sqrt{2}$ for $i = 1, 2, \ldots, 9$. This makes the $y_i$ individually standard normal but with some correlations among them.

  • Compute $w = x_1 + x_2 + \cdots + x_{10}$. Note that $w = \sqrt{2}(y_1 + y_3 + y_5 + y_7 + y_9)$.

  • Add some independent normally distributed error to $w$. With a little experimentation I found that $z = w + \varepsilon$ with $\varepsilon \sim N(0, 6)$ works pretty well. Thus, $z$ is the sum of the $x_i$ plus some error. It is also the sum of some of the $y_i$ plus the same error.

We will consider the $y_i$ to be the independent variables and $z$ the dependent variable.

Here's a scatterplot matrix of one such dataset, with $z$ along the top and left and the $y_i$ proceeding in order.

Scatterplot matrix

The expected correlations among $y_i$ and $y_j$ are $1/2$ when $|i-j|=1$ and $0$ otherwise. The realized correlations range up to 62%. They show up as tighter scatterplots next to the diagonal.

Look at the regression of $z$ against the $y_i$:

      Source |       SS       df       MS              Number of obs =      50
-------------+------------------------------           F(  9,    40) =    4.57
       Model |  1684.15999     9  187.128887           Prob > F      =  0.0003
    Residual |  1636.70545    40  40.9176363           R-squared     =  0.5071
-------------+------------------------------           Adj R-squared =  0.3963
       Total |  3320.86544    49  67.7727641           Root MSE      =  6.3967

------------------------------------------------------------------------------
           z |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
          y1 |   2.184007   1.264074     1.73   0.092    -.3707815    4.738795
          y2 |   1.537829   1.809436     0.85   0.400    -2.119178    5.194837
          y3 |   2.621185   2.140416     1.22   0.228    -1.704757    6.947127
          y4 |   .6024704   2.176045     0.28   0.783    -3.795481    5.000421
          y5 |   1.692758   2.196725     0.77   0.445    -2.746989    6.132506
          y6 |   .0290429   2.094395     0.01   0.989    -4.203888    4.261974
          y7 |   .7794273   2.197227     0.35   0.725    -3.661333    5.220188
          y8 |  -2.485206    2.19327    -1.13   0.264     -6.91797    1.947558
          y9 |   1.844671   1.744538     1.06   0.297    -1.681172    5.370514
       _cons |   .8498024   .9613522     0.88   0.382    -1.093163    2.792768
------------------------------------------------------------------------------

The F statistic is highly significant but none of the independent variables is, even without any adjustment for all 9 of them.

To see what's going on, consider the regression of $z$ against just the odd-numbered $y_i$:

      Source |       SS       df       MS              Number of obs =      50
-------------+------------------------------           F(  5,    44) =    7.77
       Model |  1556.88498     5  311.376997           Prob > F      =  0.0000
    Residual |  1763.98046    44  40.0904649           R-squared     =  0.4688
-------------+------------------------------           Adj R-squared =  0.4085
       Total |  3320.86544    49  67.7727641           Root MSE      =  6.3317

------------------------------------------------------------------------------
           z |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
          y1 |   2.943948   .8138525     3.62   0.001     1.303736     4.58416
          y3 |   3.403871   1.080173     3.15   0.003     1.226925    5.580818
          y5 |   2.458887    .955118     2.57   0.013      .533973    4.383801
          y7 |  -.3859711   .9742503    -0.40   0.694    -2.349443    1.577501
          y9 |   .1298614   .9795983     0.13   0.895    -1.844389    2.104112
       _cons |   1.118512   .9241601     1.21   0.233    -.7440107    2.981034
------------------------------------------------------------------------------

Some of these variables are highly significant, even with a Bonferroni adjustment. (There's much more that can be said by looking at these results, but it would take us away from the main point.)

The intuition behind this is that $z$ depends primarily on a subset of the variables (but not necessarily on a unique subset). The complement of this subset ($y_2, y_4, y_6, y_8$) adds essentially no information about $z$ due to correlations—however slight—with the subset itself.

This sort of situation will arise in time series analysis. We can consider the subscripts to be times. The construction of the $y_i$ has induced a short-range serial correlation among them, much like many time series. Due to this, we lose little information by subsampling the series at regular intervals.

One conclusion we can draw from this is that when too many variables are included in a model they can mask the truly significant ones. The first sign of this is the highly significant overall F statistic accompanied by not-so-significant t-tests for the individual coefficients. (Even when some of the variables are individually significant, this does not automatically mean the others are not. That's one of the basic defects of stepwise regression strategies: they fall victim to this masking problem.) Incidentally, the variance inflation factors in the first regression range from 2.55 to 6.09 with a mean of 4.79: just on the borderline of diagnosing some multicollinearity according to the most conservative rules of thumb; well below the threshold according to other rules (where 10 is an upper cutoff).

$\endgroup$
4
  • 7
    $\begingroup$ Great answer. A plus 1 from me. I would have liked to give it more. $\endgroup$ Commented May 4, 2012 at 22:11
  • $\begingroup$ Nice work! I just wonder about one specific example I ran into (on real data): logistic regression with p=10 predictors, n=500 observations (balanced response classes), largest VIF < 1.5, where one ends up with significant full model Likelihood Ratio test (at 0.0003) and all insignificant predictors (two smallest p-values at ~0.11, rest are > 0.25). I understand the main point of "It takes very little correlation among the independent variables", but in your example VIFs average 4.8, which is whopping 3.5 times what I have (avg VIF ~ 1.3), and it's still an issue... due to same reasons? $\endgroup$
    – UsDAnDreS
    Commented Jan 22, 2021 at 22:50
  • $\begingroup$ @UsDAnDreS I can't tell. In my example there's a lot of multicollinearity--my nine variables are confined almost to a space of just five dimensions. Perhaps in your case your ten variables have only one or two degrees of redundancy among them, but we would need better diagnostics than VIF to determine that. $\endgroup$
    – whuber
    Commented Jan 23, 2021 at 0:23
  • $\begingroup$ THANK YOU, THIS IMPROVED MY P-VALUE BY 72% !! $\endgroup$
    – NoName
    Commented Aug 24, 2021 at 0:38
44
$\begingroup$

Multicollinearity

  • As you note, and as has been discussed in this previous question, high levels of multicollinearity is one major cause of a statistically significant $R^2$ but statically non-significant predictors.
  • Of course, multicollinearity is not just about an absolute threshold. Standard errors on regression coefficients will increase as intercorrelations with the focal predictor increase.

Multiple almost significant predictors

  • Even if you had no multicollinearity, you can still get non-significant predictors and an overall significant model if two or more individual predictors are close to significant and thus collectively, the overall prediction passes the threshold of statistical significance. For example, using an alpha of .05, if you had two predictors with p-values of .06, and .07, then I wouldn't be surprised if the overall model had a p<.05.
$\endgroup$
1
  • $\begingroup$ Nice concise answer. To add to this, I would suggest perturbing the data (or removing a predictor) and seeing if there's is a noticeable change in the coefficients of the regression. For example, look out for sign changes. $\endgroup$ Commented Aug 12, 2015 at 6:59
43
$\begingroup$

This happens when the predictors are highly correlated. Imagine a situation where there are only two predictors with very high correlation. Individually, they both also correlate closely with the response variable. Consequently, the F-test has a low p-value (it is saying that the predictors together are highly significant in explaining the variation in the response variable). But the t-test for each predictor has a high p-value because after allowing for the effect of the other predictor there is not much left to explain.

$\endgroup$
2
  • $\begingroup$ Hi Rob, sorry for disturbing you. I read through your answer (because I am facing the question situation right now) but I can not understand what you mean by saying "after allowing for the effect of the other predictor there is not much left to explain.". May I ask you to explain that to me? Thanks a lot. $\endgroup$
    – yue86231
    Commented Jun 15, 2014 at 19:48
  • 3
    $\begingroup$ @yue86231 It means that although we have one p-value for each predictor, we can not interpret each p-value in isolation. Each predictor t-test can only show the significance of a variable after accounting for the variance explained by all other variables. The linear regression coefficients and standard error are produced at the same time, so to speak, and the two predictors reduce each other significance. $\endgroup$ Commented Aug 26, 2014 at 17:28
12
$\begingroup$

Consider the following model: $ X_1 \sim N(0,1)$, $X_2 = a X_1 + \delta$, $Y = bX_1 + cX_2 + \epsilon$, $\delta$, $\epsilon$ and $X_1$ are all mutually independent $N(0,1)$.

Then $${\rm Cov}(X_2,Y) = {\rm E}[(aX_1+\delta)(bX_1+cX_2+\epsilon)]={\rm E}[(aX_1+\delta)(\{b+ac\}X_1+c\delta+\epsilon)]=a(b+ac)+c$$

We can set this to zero with say $a=1$, $b=2$ and $c=-1$. Yet all the relations will obviously be there and easily detectable with regression analysis.

You said that you understand the issue of variables being correlated and regression being insignificant better; it probably means that you have been conditioned by frequent mentioning of multicollinearity, but you would need to boost your understanding of the geometry of least squares.

$\endgroup$
10
$\begingroup$

A keyword to search for would be "collinearity" or "multicollinearity". This can be detected using diagnostics like Variance Inflation Factors (VIFs) or methods as described inthe textbook "Regression Diagnostics: Identifying Influential Data and Sources of Collinearity" by Belsley, Kuh and Welsch. VIFs are much easier to understand, but they can't deal with collinearity involving the intercept (i.e., predictors that are almost constant by themselves or in a linear combination) - conversely, the BKW diagnostics are far less intuitive but can deal with collinearity involving the intercept.

$\endgroup$
9
$\begingroup$

The answer you get depends on the question you ask. In addition to the points already made, the individual parameters F values and the overall model F values answer different questions, so they get different answers. I have seen this happen even when the individual F values are not that close to significant, especially if the model has more than 2 or 3 IVs. I do not know of any way to combine the individual p-values and get anything meaningful, althought there may be a way.

$\endgroup$
3
  • 3
    $\begingroup$ (-1) Yes - the original poster is noting that he/she has seen it happen too. The question was what exactly are some things that may cause this other than collinearity and I don't see how this is an answer. $\endgroup$
    – Macro
    Commented Aug 19, 2011 at 10:46
  • 4
    $\begingroup$ @Macro The downvote seems a little harsh, because there is a useful and valid observation in this reply: the tests for overall significance and for individual variable significance "answer different questions." Admittedly that's qualitative, but no more so then the first answer with many upvotes; and to that answer it adds some valid intuition, arguably making it an improvement over that answer. $\endgroup$
    – whuber
    Commented Aug 19, 2011 at 14:18
  • 2
    $\begingroup$ I never said there wasn't valid information or intuition supplied by this reply. If I had a good answer for this question I'd have responded by now - it's kind of a tough question - I was only saying that this response doesn't seem to answer the question in any sense of the word. $\endgroup$
    – Macro
    Commented Aug 19, 2011 at 14:27
9
$\begingroup$

One other thing to keep in mind is that the tests on the individual coefficients each assume that all of the other predictors are in the model. In other words each predictor is not significant as long as all of the other predictors are in the model. There must be some interaction or interdependence between two or more of your predictors.

As someone else asked above - how did you diagnose a lack of multicollinearity?

$\endgroup$
5
$\begingroup$

One way to understand this is the geometry of least squares as @StasK suggests.

Another is to realize it means that X is related to Y when controlling for the other variables, but not alone. You say X relates to unique variance in Y. This is right. The unique variance in Y, though, is different from the total variance. So, what variance are the other variables removing?

It would help if you could tell us your variables.

$\endgroup$
1
$\begingroup$

While most of existing answers pointed out the multicollinearity is one common cause of this paradox, it seems that none of them demonstrated why it is so from the mathematical perspective. This answer tries to fill this gap.

To make notations less cluttered, suppose the number of regressors is $2$ and the number of observations is $n$, so the model can be written as: \begin{align} y_i = \beta_0 + \beta_1X_{1, i} + \beta_2X_{2, i} + \epsilon_i, \quad i = 1, 2, \ldots, n. \end{align}

We are interested in testing

  1. A single $\beta_i$ is $0$: \begin{align} & H_0: \beta_1 = 0 \text{ v.s. } H_1: \beta_1 \neq 0. \tag{1.1}\label{1.1} \\ & H_0: \beta_2 = 0 \text{ v.s. } H_1: \beta_2 \neq 0. \tag{1.2}\label{1.2} \end{align}

  2. All $\beta$s are 0: \begin{align} H_0: \beta_1 = \beta_2 = 0 \text{ v.s. } H_1: \beta_1 \neq 0 \text{ or } \beta_2 \neq 0. \tag{2}\label{2} \end{align}

It is well-known that the testing procedures applying to problem $\eqref{1.1}$-$\eqref{1.2}$ and $\eqref{2}$ are $t$-test and $F$-test respectively. However, the key to explain the posed paradox is recognizing the $t$-test to problem $\eqref{1.1}$-$\eqref{1.2}$ can also be viewed as an equivalent $F$-test, so that we are actually applying the partial $F$-test to problem $\eqref{1.1}$-$\eqref{1.2}$ and the overall $F$-test to problem $\eqref{2}$ respectively (a good reference to this is Applied Linear Statistical Models by Kutner et al., Section 7.3), whose testing statistics are \begin{align} F_1 = \frac{MSR(X_1|X_2)}{MSE} = \frac{SSR(X_1|X_2)}{MSE} = \frac{SSR(X_1, X_2) - SSR(X_2)}{MSE} \tag{3.1}\label{3.1} \\ F_2 = \frac{MSR(X_2|X_1)}{MSE} = \frac{SSR(X_2|X_1)}{MSE} = \frac{SSR(X_1, X_2) - SSR(X_1)}{MSE} \tag{3.2}\label{3.2} \end{align} and \begin{align} F_0 = \frac{MSR(X_1, X_2)}{MSE} = \frac{SSR(X_1, X_2)}{2MSE} \tag{4}\label{4} \end{align} respectively. At the significance level $\alpha$, $F_1$ and $F_2$ are compared with $q_1^* = F_{1, n - 3}(1 - \alpha)$, while $F_0$ is compared with $q_2^*= F_{2, n - 3}(1 - \alpha)$, to determine whether $H_0$ should be rejected.

The key concept in the test statistic $\eqref{3.1}$ is the term $SSR(X_1|X_2)$, known as sequential/extra sums of squares (see Chapter 7 of the same reference above and this link for more details), which accounts for the reduction in the error sum of squares caused by adding $X_1$ to the regression model when $X_2$ is already in the model. Thus $SSR(X_1|X_2)$ measures the marginal effect of adding $X_1$ to the regression model when $X_2$ is already in the model. Clearly, $SSR(X_2|X_1)$ bears the same interpretation. Having understood this concept, comparing $\eqref{3.1}$-$\eqref{3.2}$ against $\eqref{4}$ immediately indicates a scenario such that the overall $F$-test is significant while partial $F$-tests are insignificant, namely when $SSR(X_1, X_2)$ is big but both $SSR(X_1|X_2)$ and $SSR(X_2|X_1)$ are small (relative to $MSE$). In other words, the variation of $y$ can be adequately explained by any simple regression model where the single predictor is $X_1$ or $X_2$ (therefore $SSR(X_i)$ is big, and as a result $SSR(X_1, X_2)$ is big), but when we tried to further increase the $y$-variation explain ratio by adding the remaining variable to the existing simple regression model, the improvement is very limited (therefore both $SSR(X_1|X_2)$ and $SSR(X_2|X_1)$ are small). One example that fits this setting is multicollinearity, as described in @Rob Hyndman's answer. However, multicollinearity is not the only example that fits the general setting, as I will briefly discuss at the end of this answer.

Now let's deepen the above observation with the help of linear algebra -- my goal is to re-express $\eqref{3.1}$ and $\eqref{3.2}$ in such a manner that all sums of squares can be interpreted as vector lengths. To this end, let $y = (y_1, \ldots, y_n)' \in \mathbb{R}^n$, $e = (1, \ldots, 1)' \in \mathbb{R}^n$, $x_i = (X_{i, 1}, \ldots, X_{i, n})' \in \mathbb{R}^n, i = 1, 2$, $X = \begin{bmatrix} e & x_2 & x_1\end{bmatrix}$ (note that $x_2$ precedes $x_1$ because we let $x_2$ enter the model first). Furthermore, let $X = QR$ be the QR decomposition of $X$, where $Q = \begin{bmatrix} q_1 & q_2 & q_3\end{bmatrix}$. Following the calculation in this answer, it can be shown that \begin{align} & SSR(X_1, X_2) = y'(q_2q_2' + q_3q_3')y = \|P_{[q_2, q_3]}y\|^2, \tag{5}\label{5} \\ & SSR(X_1|X_2) = y'q_3q_3'y = \|P_{[q_3]}y\|^2. \tag{6}\label{6} \end{align} Here $P_My$ stands for the projection of $y$ onto the space $M$, and we use $[v_1, v_2, \ldots, v_m]$ to denote the space spanned by vectors $v_1, v_2, \ldots, v_m$.

Similarly, one can show that \begin{align} SSR(X_2|X_1) = y'\tilde{q}_3\tilde{q}_3'y = \|P_{[\tilde{q}_3]}y\|^2, \tag{7}\label{7} \end{align} where $\tilde{q}_3$ is the third column of the $\tilde{Q} := \begin{bmatrix} q_1 & \tilde{q}_2 & \tilde{q}_3\end{bmatrix}$ in the QR decomposition of $\tilde{X} = \begin{bmatrix} e & x_1 & x_2 \end{bmatrix} = \tilde{Q}\tilde{R}$.

With these preparations, suppose that

  1. $y$ is highly correlated with both $x_1$ and $x_2$.
  2. $x_1$ and $x_2$ are highly correlated.

Condition 1 implies that both $\|P_{[q_2]}y\|$ and $\|P_{[\tilde{q}_2]}y\|$ are big (whence $\|P_{[q_2, q_3]}y\| \geq \|P_{[q_2]}y\|$ is big), while condition 2 implies both $\|P_{[q_3]}y\|$ and $\|P_{[\tilde{q}_3]}y\|$ are small. While the first implication is straightforward, the second implication needs elaboration: without loss of generality, let's show that $\|P_{[q_3]}y\|$ is small. To this end, note that $[e, x_2] = [q_1, q_2]$ and $[e, x_2, x_1] = [q_1, q_2, q_3]$, it follows that \begin{align} \|P_{[q_3]}y\|^2 = \|P_{[e, x_2, x_1]}y\|^2 - \|P_{[e, x_2]}y\|^2 = \|P_{[e, x_2]^\perp}\left(P_{[e, x_2, x_1]}y\right)\|^2, \end{align}
where $[e, x_2]^\perp$ is the orthogonal complement of $[e, x_2]$. Under condition 2, it is easy to see that $[e, x_2, x_1] \approx [e, x_2]$, whence the projection of $P_{[e, x_2, x_1]}y$ onto $[e, x_2]^\perp$ is very close to zero, hence $\|P_{[q_3]}y\|^2$ must be small. Similarly, $\|P_{[\tilde{q}_3]}y\|^2$ is small. In summary, we showed that under condition 1 and condition 2, $\eqref{5}$ is big (hence $F_0$ in $\eqref{4}$ is big, which results in significant $F$-test outcome) and $\eqref{6}$ and $\eqref{7}$ are small (hence $F_1$ in $\eqref{3.1}$ and $F_2$ in $\eqref{3.2}$ are small, which results in insignificant $t$-test outcomes).

To conclude the discussion of the multicollinearity cause, let me point out that the mere multicollinearity without high correlation between $y$ and predictors would not guarantee the significant $F$-test (i.e., the lack of condition 1 above). Many existing answers attributed the paradox to (oversimplified) multicollinearity, which is technically insufficient (remember that multicollinearity is a property of the design matrix $X$ only that has nothing to do with the response).

Another cause of this paradox is as described by the second reason of Jeromy Anglim's answer, and can also be explained using the framework built earlier. This actually corresponds to the scenario \begin{align} & \max(\|P_{[q_3]}y\|^2, \|P_{[\tilde{q}_3]}y\|^2) \leq MSE \cdot q_1^*, \\ & \|P_{[q_2]}y\|^2 + \|P_{[q_3]}y\|^2 > 2MSE \cdot q_2^*. \tag{8}\label{8} \end{align} Since we have the freedom of controlling the magnitudes of $P_{[q_2]}y$ and $P_{[q_3]}y$ (from the counterexample construction perspective), for $q_1^* < 2q_2^*$, there are countless concrete examples such that $\eqref{8}$ holds.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.