I am trying to fit a model to some observed data with the least squares method. Now, I am at the stage where I have run a stepwise regression (traditional), with Entry level $=0.025$ and Stay level $=0.025$ (I really don't know what thresholds to use, this is my first model I am trying to fit).

Anyhow, I am getting a pretty good fit - if one should rely on $R^2 = 0.98%$. However I can see from scatterplots for the remaining $5$ predictors that there seem to be some correlation between some of the predictors. So, I decided to check Pearsons coefficients ($\rho$) which gave the result that the predictors $x_1$ and $x_2$ ($\rho=0.98%$, P value $<10^{-3}$ ), $x_3$ and $x_4$ ($\rho=0.94%$, P value < $10^{-3}$ )were strongly correlated.

My questions are:

  1. Is it possible for two variables that have so strong correlation to survive in a stepwise regression?
  2. How do I proceed from here? How do I know which of the predictors to omit?

(If more information is needed in order to give a good answer please let me know so that i can give that information).

  • $\begingroup$ Can you explain what SLENTRY=0.025 and SLSTAY=0.025 means? $\endgroup$ – Jeremy Miles Nov 25 '14 at 0:23
  • $\begingroup$ i've changed it .... $\endgroup$ – Danny Nov 25 '14 at 0:28
  • $\begingroup$ I still don't know what it means. Is that a p-value? $\endgroup$ – Jeremy Miles Nov 25 '14 at 0:52
  • $\begingroup$ They can survive because the data say that they can survive (based on your criteria, which is p-value, I think?). You are using an automatic algorithm to build a model. If you think you should use an automatic algorithm it's a bit strange to not like the result and remove a variable. $\endgroup$ – Jeremy Miles Nov 25 '14 at 0:53

It seems like you are violating the assumption of multicollinearity. This implies that your predictor variables are too highly correlated and you must do something about it, otherwise the estimates computed for the predictors are not accurate.

There are several things you can do to tackle this problem. I would assume that variables correlated as high as 0.98 pretty much measure the same thing, so it might be up to you to leave one out over the other one. As a rule of thumb, you should generally avoid having predictors in the model with a correlation higher than 0.7. You might also want to calculate the VIF and tolerance values for your predictors to make sure which ones are causing issues in your model. Another idea would be to maybe run a factor analysis for your predictor variables and use these composite variables (factors) as predictors.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.