2
$\begingroup$

I've looked through a related question and another on this topic but it seems I'm still missing something. I understand that in a multiple regression from these questions and answers that a given parameter is estimated while other variables are held at their mean.

If I had a variable Y that was predicted by a series of variables 1 and ran a multiple linear regression, would the p-values for the parameter estimates equal the p-values of the paired Pearson's product-moment correlation.

Eg. Does the p-value of the Pearson correlation between X₁ and Y equal the p-value of the parameter estimate of X₁ in the linear regression

2

My intuition is that this is not correct. The p-values would be the same if estimation of a parameter was done while other variables were held constant at 0, but they are not, they're held at their mean?

Therefore, the two should be equal if we were talking of a simple linear regression where:

3

but does this hold true when there are multiple predicting values, or does the p-value of a parameter estimate in a multiple regression depend on the other variables in the equation?

$\endgroup$
0

1 Answer 1

2
$\begingroup$

The p-value does not have to remain the same in the simple linear regression and the multiple linear regression case. A common reason for this is collinearity between predictive and non-predictive features.

Say $X_1$ is not predictive of $Y$, but is correlated with a highly predictive feature, $X_2$. For instance, $Y$ could be a final exam score, $X_2$ could be number of hours studied and $X_1$ could be number of study breaks taken. Clearly, $X_1$ is not directly predictive of $Y$: taking more study breaks doesn't improve test scores. However, it is correlated with study time. Therefore, regressing $Y$ onto $X_1$ can yield a small, significant p-value. This is because $X_1$ acts as a surrogate of $X_2$ while $X_2$ is absent from the model.

However, the tables turn when regressing $Y$ onto both $X_1$ and $X_2$. The only reason $X_1$ was predictive before was because it was correlated with $X_2$, but now when $X_2$ is held fixed, $X_1$'s predictive capacity is gone. Continuing with the example: this states that if I studied 2 hours, taking more (or less) study breaks does not affect my final exam score. Therefore, $X_1$ receives a large, insignificant p-value.

So $X_1$ can have both small and large p-values, depending on the other features used in the model

$\endgroup$
1
  • $\begingroup$ Thanks for your explanation. Could this work the same way where two variables significantly predict y when the are regressed independently of one another but when they are both used to predict Y in a multiple linear regression, neither are significant because of high covariance between the two? Also, is the p-value from a Pearson's cor the same as a correlation from a simple regression though? $\endgroup$
    – hamilthj
    Commented Mar 30, 2017 at 16:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.