1
$\begingroup$

Let's say you have 8 variables in a regression model. If some of them are correlated, what degree or percent of correlation should you considering removing some of the variables from the equation? How about covariance - what effect does that have?

Lastly, how many variables are too many to inform the model? 5, 15, 50, 500? The model I have is a profit/financial type model and is fairly straightforward - profit is the output.

Improve this question
$\endgroup$
4
  • $\begingroup$ These are very difficult questions to answer without more context and knowing exactly what you want to do with your model. Correlation can cause problems by making some of the computational methods difficult (e.g. taking inverse of matrix) for doing regression. So again, whether or not you should remove variables depends on what you are trying to do with your model $\endgroup$
    – bdeonovic
    Oct 31, 2013 at 23:00
  • $\begingroup$ Hi Sir, thanks for your response. Its nothing too fancy. Let me explain a bit more about my model. The model is simply a list of paramters or variables related to a NYSE stock market price or other financial piece of information. My goal is to predict stock prices and profits by varying the variables within the reg. equation. My stat skills are not great, so please enlighten me. $\endgroup$
    – Matthew C
    Oct 31, 2013 at 23:18
  • $\begingroup$ Except in respect of the emphasis on correlation/multicollinearity, this question is at the least a near-duplicate of stats.stackexchange.com/questions/74140 ... whether you want to remove variables at all depends on whether your goal is statistical inference or predictive power. $\endgroup$
    – Silverfish
    Nov 1, 2013 at 0:05
  • $\begingroup$ Hi, sir, can you explain, what is the difference between statistical inference and predictive power? $\endgroup$
    – Matthew C
    Nov 2, 2013 at 0:40

1 Answer 1

Reset to default
1
$\begingroup$

This is called Multicollinearity in regression. Dropping is just one way to handle this situation and it is not the only remedy. Sometimes you standardize your independent variables, or get more data (if possible), dropping a term is another option, or even fitting a regression with the knowledge of having Multicollinearity! To assess how much they are correlated, you need to find a factor called Variance inflation factor (VIF), where $VIF(\hat{\beta_j})=\dfrac{1}{1-R^2_i}$ and $R^2_i$ is the coefficient of determination of the regression model, i.e. you replace your $Y$ (dependent variable) with $X_i$ and fit your regression model with all other independent variables. Then you find your $R^2$ as usual. The result is denoted by $R^2_i$ to emphasize the fact that you regressed $X_i$ against other independent variables. As a rule of thumb when $VIF(\hat{\beta_j})>5$ (or in some references $VIF(\hat{\beta_j})>10$), we say that multicollinearity is high.

To answer your 2nd question, just think about the relation between covariance and correlation of two random variables. Then I guess you can find your answer. the correlation is just scaled version of covariance.

For the last question: I don't think there is specific number of variables to call "too many" that works in any model. The thing is that you need to fit a good regression model but it does not mean that you should add too many variables. Because over-parametrization is another problem that should be avoided.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Okay, let me figure out the correlation in Excel and ill get back to you. Not sure I totally understood what you typed, but hang on lol. $\endgroup$
    – Matthew C
    Oct 31, 2013 at 23:33
  • 1
    $\begingroup$ Another thing to consider is using Ridge Regression, which was specifically designed to deal with multicollinearity. Or perhaps some other penalized regression approach (lasso, Enet, Mnet, SCAD, etc.) $\endgroup$
    – bdeonovic
    Nov 1, 2013 at 12:05
  • $\begingroup$ Hi sir, thanks for the advice. I did the correlation in Excel with the CORREL function and it worked pretty well. Some of the variables have a 1 to 1, some have 0 and others are about .40 to .45. Obviously I should remove the 1.0 correlation, but what about the variables that have a .40 to .45, is that too high? $\endgroup$
    – Matthew C
    Nov 2, 2013 at 0:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.