1
$\begingroup$

I have a multilinear regression with two x variables. Here are my key values:

  • R square = .407
  • Adjusted R square = .370
  • Observations = 36
  • Significance of F -s super low (.00018)
  • x1 coef = -.0021; tstat = .013; P-value = .893
  • x2 coef = 1.06; tstat = .539; P-value = .593

I think this means that:

  • something is going on (reject null) with a super low F significance
  • about 40% of the change in Y is explained by the two variables
  • x2 has a greater probable (more significance) influence on y with its much higher tstat.
  • x2 has a greater magnitude of influence on y with its higher coef (scaling comes into play here).

Do the P-values mean that the confidence in the significance is 11% for x1 and 41% for x2, and therefore x1 should certainly be ignored, and maybe x2 as well?

$\endgroup$
  • $\begingroup$ Calculate correlation of x1 and x2... $\endgroup$ – Glen_b Sep 21 '17 at 5:28
  • $\begingroup$ Ok, I did used =correl(a1:a36,b1:b36) and got 0.9965378375. So I assume that means these values are highly correlated. :^) $\endgroup$ – JoeD Sep 21 '17 at 21:30
  • $\begingroup$ How I guessed that: A high F statistic (/low p-value) indicates you're getting predictive value above randomness (something is "fitting"), while high p-values for each term suggests -- after adjusting for the other predictor -- there's not much "left over". This happens when the variables are highly correlated; the first picks up whatever there is and the other doesn't say much the first did not. Looked at another way, the large correlation inflates each of the coefficient's standard errors, but the model is still predicting a substantial proportion of the variation in the response. $\endgroup$ – Glen_b Sep 21 '17 at 22:28
  • $\begingroup$ Thanks Glen. FYI - x1 is annual atmospheric CO2 content (Mauna Loa), and x2 is the earth's angle to the sun (Obliquity). Y is annual average temperature (as measured by satellite). $\endgroup$ – JoeD Sep 22 '17 at 6:26
2
$\begingroup$

The $F$ statistic is directly related to the total $R^2$ of the regression. So if your covariates, together, explain a lot of the variation of $y$, then you will have a high $F$ statistics.

But the variance of each specific coefficient depends on how the covariates are correlated. To see how that happens, let's consider a linear regression with two covariates $X= [x_1, x_2]$:

$$ y = \hat{\beta}_1x_1 + \hat{\beta}_2x_2 + \hat{\epsilon} $$

To simplify computation, let the variables be standardized to have unit variance and mean zero. Let $\hat{\beta} = [\beta_1, \beta_2]$. The estimated variance of $\hat{\beta}$ is:

$$ \begin{align} \widehat{Var(\hat{\beta})} &= (X'X)^{-1}\hat{\sigma} \\ &= Cor(X)^{-1} \frac{SSR}{(n-1)(n-2)}\\ &=Cor(X)^{-1} \frac{(1 - R^2)}{n-2}\\ \end{align} $$

Where $SSR$ is the sum of squared residuals, $Cor(X)$ is the correlation matrix of $X$ and $R^2$ is the $R^2$ of the whole regression. We are interested in the diagonal elements of this variance, so let's pick the first element of the diagonal, which is $\widehat{Var(\hat{\beta}_1)}$. Since our variables are standardized, the first term of $Cor(X)^{-1}$ is simply $\frac{1}{1 - Cor(x_1, x_2)^2}$. Hence:

$$ \begin{align} \widehat{Var(\hat{\beta}_1)} &= \frac{(1 - R^2)}{(1 - Cor(x_1, x_2)^2)(n-2)}\\ &= \frac{(1 - R^2)}{(1 - R^2_{x_1 \sim x_2})(n-2)} \end{align} $$

Where $R^2_{x_1 \sim x_2}$ is the percentage of the variance of $x_1$ explained by $x_2$. That is, while a large $R^2$ for the whole regression reduces variances overall, if $x_2$ explains a lot of the variance of $x_1$ this can have serious effects on the variance of the coefficient, giving you high p-values.

So your suggested interpretation is not correct. What your results mean is that $\{x_1, x_2\}$ explain a lot of the variation of $y$ together, but since the correlation between them is high, you can't estimate the contribution of each term precisely with your sample size.

(As a side note, it seems you are interpreting p-values as posterior probabilities --- but that would be another whole answer on its own, you should take a look at some of the answers here and here. And if you are trying to use regression for causal inference, you should take a look here.)

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ From the comment in the question above and your answer, I understand the correlation is high (thanks, Glen and Carlos!). So if x1 and x2 aren't independent, then we can't know the impact of each discreetly. But doesn't t-stat indicate anything in x2's favor? $\endgroup$ – JoeD Sep 21 '17 at 21:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.