0
$\begingroup$

If I given a model with 3 variables($X_1, X_2$ and $ X_3$) and a correlation between them are not high. The highest correlation coefficient from the correlation matrix is equal to $0.699614004$ and is between $X_2$ and $X_3$. Is this coefficient high enough that to drop the variable $X_2$ in order for the model to be precise?

Generally how to know when to drop a variable form the model?

$\endgroup$
4
  • 2
    $\begingroup$ This is too abstract and over summarized to answer. What are these variables, and what do they measure? What are your goals with the model? $\endgroup$ Mar 29 '18 at 16:45
  • $\begingroup$ It is a model where Y is performance IQ, and $X_1, X_2,X_3$ are the Brain weight, height and weight of the person. I am asked to determine whether any variables have to be dropped basing on the correlation matrix and the adjusted $R^2$ coefficient , which is $0.2$. Since it is not high I assume I need to drop some variables, however, their correlation coefficients are not very high(the highest one is $0.6996$ and $0.588$). How can I determine which variables should I drop? $\endgroup$
    – user200918
    Mar 29 '18 at 17:04
  • 3
    $\begingroup$ Why drop any? Why are you fitting a model? $\endgroup$ Mar 29 '18 at 17:09
  • 2
    $\begingroup$ What Scortchi said. Is the intent of the model to learn about the effect of one of the predictors, to make predictions, something else? $\endgroup$ Mar 29 '18 at 18:09
2
$\begingroup$

What you are really asking is: Should I worry about collinearity among the predictors in my model? Collinearity refers to a situation where two or more of the predictors in a regression model are moderately or highly correlated. (Collinearity is also referred to as multicollinearity.)

As other people have already pointed out, whether or not you need to worry about collinearity ultimately depends on the purpose of your model. Usually, if you are interested in using your model to make predictions, collinearity is not as worrisome as if you intend to use the model to learn something about the effect of the predictor variable engaged in collinearity on the outcome variable. The extent of collinearity also factors into whether or not you need to worry about collinearity.

This blog post provides a nice description of when you need to address collinearity and when you don't: http://statisticsbyjim.com/regression/multicollinearity-in-regression-analysis/.

$\endgroup$
1
$\begingroup$

Generally speaking, $R^2$ always increases as you increase the number of variables in your model, so by itself it is not a good criterion to know when you should stop adding variables. Instead you should use a different number that in some sense measures that "higher $R^2$ is better, but you don't want too many variables either". Quantities that measure this are the adjusted $R^2$, the AIC, and the BIC.

In your specific situation, since you have two variables with moderately high correlation, I would also try some sort of dimensionality reduction algorithm, and they try a linear regression on just two variables. I would look into PCA for its simplicity.

$\endgroup$
1
  • 2
    $\begingroup$ I just don't think this question is answerable yet. Your advice appliances in a particular set of circumstances, but it's not yet clear that the asker is in that set of circumstances. $\endgroup$ Mar 29 '18 at 18:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy