In a linear regression model, if some of the predictors are correlated, then in the output of most software, you will see very large p-value in those coefficients and very high standard error. My question is: does this matter in terms of the prediction?

I made a small example myself. I create the following dataset

$Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \epsilon$, where $\epsilon\sim~N(0,1)$ which is independent of $X_1,X_2$ and $X_2 = aX_1+b$, i.e., $X_1$ and $X_2$ are perfectly correlated. So after rearrangement, we see that $Y=\beta_0+b\beta_2 + X_1(\beta_1+a\beta_2)+\epsilon$. Then I use R to run linear regression. Not surprise, R reports NA for the coefficient, standard error and p value of $X_2$. But then I look at the estimated coefficient of $X_1$, it is very close to $\beta_1+a\beta_2$ and the estimated intercept is also very close to $\beta_0+b\beta_2$.

We all know that the estimator of the coefficient $\theta=[\beta_0,\beta_1,\beta_2]$ has the form $\hat{\theta}=(X^{T}X)^{-1}X^TY$. In this case the matrix $X^{T}X$ is not invertible. I wonder how R reports correct estimate for $\beta_1+a\beta_2$ and $\beta_0+b\beta_2$ in this case? Moreover, if I only care about prediction and don't care about interpretation of the model, can I just ignore those predictors that has NA or big standard error of the coefficients estimator and just use the remaining model?

A more general question, if I made my data such that $x_2$ and $x_1$ has some correlation but not perfect, then will the LS method be able to derive a consistent estimator of the coefficients? If it can't then will there be a method achieve this goal?

  • $\begingroup$ I think you may benefit by splitting up your questions into more than one post. There are five questions which are not necessarily dependent on each other. You may get better answers when you stay focused. $\endgroup$ – Richard Hardy Feb 17 '15 at 18:38
  • $\begingroup$ Slightly off-topic, but Dave Giles has a nice (but longish) blog post on multicollinearity. $\endgroup$ – Richard Hardy Feb 17 '15 at 18:59

Your question in the first paragaph has been asked and answered before at Cross Validated. You will find answers by searching for prediction and multicollinearity. See e.g. here. If the answers are not sufficient, try to explain what exactly you are missing.

Answering the first question in your last paragraph, OLS assumptions do allow for imperfect multicollinearity and the estimation will be consistent. Depending on the degree of correlation, it may take a very large sample for this consistency to take over the effect of multicollinearity. So in practice it remains a problem when the correlation is close to perfect.

I cannot answer the two questions in the penultimate paragraph nor your last question.

  • $\begingroup$ @Hardy Correct me if I am wrong. For example, the data I cooked, among the predictors, there could be very high correlation (not perfect). Then I use OLS to fit a sample data. It is likely that the standard error of my estimator is very high and the p value is very high. But if I have enough data, i.e., very large sample size, eventually, the standard error of my estimator will go down and so is my p-value. Theoretically speaking, when sample size is infinite, I will still be able to reach the true value of those parameters in my model. $\endgroup$ – KevinKim Feb 19 '15 at 21:36
  • $\begingroup$ I think this is correct. $\endgroup$ – Richard Hardy Feb 20 '15 at 8:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.