I'm fitting a multiple linear regression model between 4 categorical variables (with 4 levels each) and a numerical output. My dataset has 43 observations.

Regression gives me the following $p$-values from the $t$-test for every slope coefficient: $.15, .67, .27, .02$. Thus, the coefficient for the 4th predictor is significant at $\alpha = .05$ confidence level.

On the other hand, the regression gives me a $p$-value from an overall $F$-test of the null hypothesis that all my slope coefficients are equal to zero. For my dataset, this $p$-value is $.11$.

My question: how should I interpret these results? Which $p$-value should I use and why? Is the coefficient for the 4th variable significantly different from $0$ at the $\alpha = .05$ confidence level?

I've seen a related question, $F$ and $t$ statistics in a regression, but there was an opposite situation: high $t$-test $p$-values and low $F$-test $p$-value. Honestly, I don't quite understand why we would need an $F$-test in addition to a $t$-test to see if linear regression coefficients are significantly different from zero.

  • 2
    $\begingroup$ If you have 4 categorical variables with 4 levels each, you should have 3*4=12 coefficients for your independent variables (plus the intercept)... $\endgroup$
    – boscovich
    Commented Mar 15, 2012 at 20:01
  • $\begingroup$ @andrea: I've decided to treat them as numerical variables. $\endgroup$
    – Leo
    Commented Mar 15, 2012 at 20:12
  • 4
    $\begingroup$ 0.02 is barely significant (especially if you consider the fact that you have five tests in total) and 0.11 is not very high. A generous interpretation would be that with a little more power the overall F-test would also be significant (and perhaps the first coefficient as well). A more conservative interpretation is that you shouldn't have much confidence in any of these results (including the coefficient with a .02 p value). Either way, you shouldn't read too much in the difference between .02 and .11. $\endgroup$
    – Gala
    Commented Mar 15, 2012 at 20:34
  • 4
    $\begingroup$ For a discussion of the opposite case, you can also see here: how can a regression be significant yet all predictors be non-significant, in addition to the question linked above. $\endgroup$ Commented Sep 13, 2012 at 15:17

4 Answers 4


I'm not sure that multicollinearity is what's going on here. It certainly could be, but from the information given I can't conclude that, and I don't want to start there. My first guess is that this might be a multiple comparisons issue. That is, if you run enough tests, something will show up, even if there's nothing there.

One of the issues that I harp on is that the problem of multiple comparisons is always discussed in terms of examining many pairwise comparisons—e.g., running t-tests on every unique pairing of levels. (For a humorous treatment of multiple comparisons, look here.) This leaves people with the impression that that is the only place this problem shows up. But this is simply not true—the problem of multiple comparisons shows up everywhere. For instance, if you run a regression with 4 explanatory variables, the same issues exist. In a well-designed experiment, IV's can be orthogonal, but people routinely worry about using Bonferroni corrections on sets of a-priori, orthogonal contrasts, and don't think twice about factorial ANOVA's. To my mind this is inconsistent.

The global F test is what's called a 'simultaneous' test. This checks to see if all of your predictors are unrelated to the response variable. The simultaneous test provides some protection against the problem of multiple comparisons without having to go the power-losing Bonferroni route. Unfortunately, my interpretation of what you report is that you have a null finding.

Several things mitigate against this interpretation. First, with only 43 data, you almost certainly don't have much power. It's quite possible that there is a real effect, but you just can't resolve it without more data. Second, like both @andrea and @Dimitriy, I worry about the appropriateness of treating 4-level categorical variables as numeric. This may well not be appropriate, and could have any number of effects, including diminishing your ability to detect what is really there. Lastly, I'm not sure that significance testing is quite as important as people believe. A $p$ of $.11$ is kind of low; is there really something going on there? maybe! who knows?—there's no 'bright line' at .05 that demarcates real effects from mere appearance.


I would like to suggest that this phenomenon (of a non-significant overall test despite a significant individual variable) can be understood as a kind of aggregate "masking effect" and that although it conceivably could arise from multicollinear explanatory variables, it need not do that at all. It also turns out not to be due to multiple comparison adjustments, either. Thus this answer is adding some qualifications to the answers that have already appeared, which on the contrary suggest that either multicollinearity or multiple comparisons should be looked at as the culprits.

To establish the plausibility of these assertions, let's generate a collection of perfectly orthogonal variables--just as non-collinear as possible--and a dependent variable that explicitly is determined solely by the first of the explanands (plus a good amount of random error independent of everything else). In R this can be done (reproducibly, if you wish to experiment) as

p <- 5 # Number of explanatory variables
x <- as.matrix(do.call(expand.grid, lapply(as.list(1:p), function(i) c(-1,1))))
y <- x[,1] + rnorm(2^p, mean=0, sd=2)

It's unimportant that the explanatory variables are binary; what matters is their orthogonality, which we can check to make sure the code is working as expected, which can be done by inspecting their correlations. Indeed, the correlation matrix is interesting: the small coefficients suggest y has little to do with any of the variables except the first (which is by design) and the off-diagonal zeros confirm the orthogonality of the explanatory variables:

> cor(cbind(x,y))
     Var1  Var2  Var3   Var4  Var5      y
Var1 1.00 0.000 0.000  0.000  0.00  0.486
Var2 0.00 1.000 0.000  0.000  0.00  0.088
Var3 0.00 0.000 1.000  0.000  0.00  0.044
Var4 0.00 0.000 0.000  1.000  0.00 -0.014
Var5 0.00 0.000 0.000  0.000  1.00 -0.167
y    0.49 0.088 0.044 -0.014 -0.17  1.000

Let's run a series of regressions, using only the first variable, then the first two, and so on. For brevity and easy comparison, in each one I show only the line for the first variable and the overall F-test:

>temp <- sapply(1:p, function(i) print(summary(lm(y ~ x[, 1:i]))))

#              Estimate Std. Error t value Pr(>|t|)   
1  x[, 1:i]       0.898      0.294    3.05   0.0048 **
F-statistic: 9.29 on 1 and 30 DF,  p-value: 0.00478 

2  x[, 1:i]Var1    0.898      0.298    3.01   0.0053 **
F-statistic: 4.68 on 2 and 29 DF,  p-value: 0.0173 

3  x[, 1:i]Var1   0.8975     0.3029    2.96   0.0062 **
F-statistic: 3.05 on 3 and 28 DF,  p-value: 0.0451 

4  x[, 1:i]Var1   0.8975     0.3084    2.91   0.0072 **
F-statistic: 2.21 on 4 and 27 DF,  p-value: 0.095 

5  x[, 1:i]Var1   0.8975     0.3084    2.91   0.0073 **
F-statistic: 1.96 on 5 and 26 DF,  p-value: 0.118

Look at how (a) the significance of the first variable barely changes, (a') the first variable remains significant (p < .05) even when adjusting for multiple comparisons (e.g., apply Bonferroni by multiplying the nominal p-value by the number of explanatory variables), (b) the coefficient of the first variable barely changes, but (c) the overall significance grows exponentially, quickly inflating to a non-significant level.

I interpret this as demonstrating that including explanatory variables that are largely independent of the dependent variable can "mask" the overall p-value of the regression. When the new variables are orthogonal to existing ones and to the dependent variable, they will not change the individual p-values. (The small changes seen here are because the random error added to y is, by accident, slightly correlated with all the other variables.) One lesson to draw from this is that parsimony is valuable: using as few variables as needed can strengthen the significance of the results.

I am not saying that this is necessarily happening for the dataset in the question, about which little has been disclosed. But knowledge that this masking effect can happen should inform our interpretation of the results as well as our strategies for variable selection and model building.

  • $\begingroup$ +1, I agree w/ this analysis. FWIW, this is the explanation I was hinting at (perhaps not well) in my discussion about power in my answer to the other question. I do have 1 question about your version here, why do you use 32 as the mean of your error term? Is that a typo, or is it important in some way? $\endgroup$ Commented Aug 21, 2012 at 20:59
  • $\begingroup$ @gung Where do you see 32? If you're referring to rnorm(2^p, sd=2), please note that the first argument is the number of terms, not the mean. The mean by default is zero and therefore has not been explicitly specified. $\endgroup$
    – whuber
    Commented Aug 21, 2012 at 21:23
  • $\begingroup$ Oh, sorry. I guess I was confusing rnorm() w/ $\mathcal N(\mu, \sigma)$. $\endgroup$ Commented Aug 21, 2012 at 21:52
  • $\begingroup$ @gung I am grateful for the opportunity to clarify the code and therefore have edited the offending line. $\endgroup$
    – whuber
    Commented Aug 21, 2012 at 21:58
  • $\begingroup$ If you find 42 instead of 32, contact Doug Adams ASAP! $\endgroup$
    – MSIS
    Commented Aug 30, 2023 at 17:24

You frequently have this happen when you have a high degree of collinearity among your explanatory variables. The ANOVA F is a joint test that all the regressors are jointly uninformative. When your Xs contain similar information, the model cannot attribute the explanatory power to one regressor or another, but their combination can explain much of the variation in the response variable.

Also, the fact that you seem to be treating you categorical variables as if they were continuous may be problematic. You are explicitly imposing restrictions like bumping $x_{1}$ from 1 to 2 has the same effect on $y$ as bumping it from 3 to 4. Sometime's that's OK, but often it's not.

  • $\begingroup$ If collinearity is a problem, then you will have high standard errors and perhaps implausibly large coefficients, maybe even with the wrong signs. To make sure that this is what is happening, calculate the variance inflation factors (VIFs) after your regression. A reasonable rule of thumb is that collinearity is a problem if the largest VIF is greater than 10. If so, you really have two options here. One is to re-specify the model to reduce the near-linear dependence by dropping some of your variables. The second is to get a larger and/or better (less homogenous) sample. $\endgroup$
    – dimitriy
    Commented Mar 15, 2012 at 20:36
  • 1
    $\begingroup$ (+1) This explanation is a good one, but it is unnecessary to attribute the phenomenon to multicollinearity: the key distinction is between jointly informative and individually informative. Including additional uncorrelated regressors (which avoids any multicollinearity) lowers the former while leaving the latter unchanged. $\endgroup$
    – whuber
    Commented Aug 22, 2012 at 16:44
  • 1
    $\begingroup$ Your answer is for the opposite situation: significant $F$ test outcome but insignificant $t$ test. How does it answer OP's question? $\endgroup$
    – Zhanxiong
    Commented May 13, 2023 at 13:07

The example in whuber's answer is very to the point (+1), to which I want to elaborate the rationale behind it from the theoretical perspective. For a better exposition, suppose the number of regressors is $2$ and the number of observations is $n$, so the model can be written as: \begin{align} y_i = \beta_0 + \beta_1X_{1, i} + \beta_2X_{2, i} + \epsilon_i, \quad i = 1, 2, \ldots, n. \end{align}

Further suppose the relation among $y = (y_1, \ldots, y_n)$, $X_1 = (X_{1,1}, \ldots, X_{1, n})$, $X_2 = (X_{2, 1}, \ldots, X_{2, n})$ is as set up by whuber: $X_1$ and $X_2$ are orthogonal, $y$ and $X_1$ are highly correlated, $y$ and $X_2$ are (almost) uncorrelated. We are interested in testing

  1. The single $\beta_1$ is $0$: \begin{align} H_0: \beta_1 = 0 \text{ v.s. } H_1: \beta_1 \neq 0. \tag{1} \end{align}

  2. All $\beta$s are 0: \begin{align} H_0: \beta_1 = \beta_2 = 0 \text{ v.s. } H_1: \beta_1 \neq 0 \text{ or } \beta_2 \neq 0. \tag{2} \end{align}

It is well-known that the testing procedures applying to problem $(1)$ and $(2)$ are $t$-test and $F$-test respectively. However, the key to explain the posed paradox is recognizing the $t$-test to problem $(1)$ can also be viewed as an equivalent $F$-test, so that we are actually applying the partial $F$-test to problem $(1)$ and the overall $F$-test to problem $(2)$ respectively (a good reference to this is Applied Linear Statistical Models by Kutner et al., Section 7.3), whose testing statistics are \begin{align} F^{(1)} = \frac{MSR(X_1|X_2)}{MSE} = \frac{SSR(X_1, X_2) - SSR(X_2)}{MSE} \tag{3} \end{align} and \begin{align} F^{(2)} = \frac{MSR(X_1, X_2)}{MSE} = \frac{SSR(X_1, X_2)}{2MSE} \tag{4} \end{align} respectively. At the significance level $\alpha$, $F^{(1)}$ and $F^{(2)}$ are compared with critical points (($1 - \alpha$)-$F$-quantiles) $q_1^* = F_{1, n - 3}(1 - \alpha)$ and $q_2^*= F_{2, n - 3}(1 - \alpha)$ respectively to determine whether $H_0$ should be rejected.

Under this specific setting, it is clear that $SSR(X_2) \approx 0$, which implies that $F^{(1)}$ is approximately $2$ times of $F^{(2)}$, however, the ratio of $q_1^*$ and $q_2^*$ is less than $2$, making the null hypothesis in $(1)$ is more likely to be rejected than the null hypothesis in $(2)$: for example, if $F^{(1)}$ is slightly greater than $q_1^*$, hence the null hypothesis in $(1)$ is (barely) rejected, then $F^{(2)} \approx 0.5F^{(1)}$ is slightly greater than $0.5q_1^*$, which will be less than $q_2^*$, as it is supposed to be considerably greater than $0.5q_1^*$. Hence the null hypothesis in $(2)$ cannot be rejected at the same significance level $\alpha$.

Now let's replicate whuber's example to illustrate the above point. As can be read from the output below, $F^{(1)} = 3.012^2 = 9.072 > q_1^* = 7.597663$, $F^{(2)} = 4.684 < q_2^* = 5.420445$, hence at $\alpha = 0.01$, $H_0$ in $(1)$ is rejected but in $(2)$ is not rejected. The reason is that the ratio of $F^{(1)}$ and $F^{(2)}$ is $F^{(1)}/F^{(2)} = 1.94$, which is considerably greater than the ratio of cutoff points $q_1^*/q_2^* = 1.40$.

> set.seed(17)
> p <- 5 # Number of explanatory variables
> x <- as.matrix(do.call(expand.grid, lapply(as.list(1:p), function(i) c(-1,1))))
> y <- x[,1] + rnorm(2^p, mean=0, sd=2)
> X <- as.data.frame(x)
> m <- lm(y ~ Var1 + Var2, X)
> summary(m)

lm(formula = y ~ Var1 + Var2, data = X)

     Min       1Q   Median       3Q      Max 
-3.13861 -1.34150  0.09369  1.10478  2.85457 

            Estimate Std. Error t value Pr(>|t|)   
(Intercept)   0.5185     0.2980   1.740  0.09244 . 
Var1          0.8975     0.2980   3.012  0.00534 **
Var2          0.1624     0.2980   0.545  0.58982   
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 1.686 on 29 degrees of freedom
Multiple R-squared:  0.2442,    Adjusted R-squared:  0.192 
F-statistic: 4.684 on 2 and 29 DF,  p-value: 0.01726

> qf(0.99, 1, 29)
[1] 7.597663
> qf(0.99, 2, 29)
[1] 5.420445

To illustrate the "insignificance of overall $F$-test inflation" effect as the number of regressors $p$, consider the same scenario as above but now $p$ ranges from $2$ to $29$ (the penultimate maximum number of regressors that $F$-test works fine).

q1 <- qf(0.99, 1, 29:2)
q2 <- qf(0.99, 2:29, 29:2)
F1 <- q1 + 0.5 # This simulates the case that t-test barely rejects H0.
F2 <- F1/(2:29) # F2 is approximately 1/p of F1 by setting.
plot(2:29, q2, xlab = "# of predictors", ylab = "")
points(2:29, q1, pch = 2)
lines(2:29, F1)
lines(2:29, F2, lty = "dashed")
       c("t-test threshold (q1)",
         "F-test threshold (q2)", 
         "t-test stat (F1)", 
         "F-test stat (F2)"), 
       pch = c(2, 1, NA, NA),
       lty = c(NA, NA, "solid", "dashed"))

The graph is as follows. It can be seen that the dashed line is always below thresholds (hence $F$-tests are always insignificant), and as $p$ increases, the gap between the dots and the dashed line also increases, indicating that the $F$-test becomes more insignificant.

enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.