I have data with five predictor variables.I want to determine which of the first five variables affect my response variable by constructing some kind of a statistical model in R. Now, i have checked for the assumptions and managed to fit (full) multiple linear model. Now, i am not so sure of how i go about the process of variable selection in R and which best method to use and how.

  • $\begingroup$ What is your goal? Prediction or inference? If inference, what are the hypotheses you want to test? $\endgroup$ – Roland Oct 7 '15 at 12:07
  • $\begingroup$ This may possibly help you. $\endgroup$ – Antoni Parellada Oct 7 '15 at 12:56
  • $\begingroup$ No, don't do stepwise selection. It has been discussed many times on this site why stepwise selection should be avoided. $\endgroup$ – Roland Oct 7 '15 at 13:11
  • $\begingroup$ Prediction because i want to test how changes in the variables for example- decreasing per capita crime rate and improvement in pupil to teacher ratio will affect the price of a property being sold in that area after these upgrades. $\endgroup$ – mapiye Oct 7 '15 at 13:17
  • 1
    $\begingroup$ You could use a shrinkage method such as the LASSO. $\endgroup$ – Roland Oct 7 '15 at 14:55

I think your queation does not has one answer. I am sure that different person will follow different approach. In my opinion, variable selection is more of art and this is where experience plays vital role.

For variables selection/inclusion in regression model, I keep track of these three things - overfitting, multicollinearity, and prediction accuracy on test data. If you dont have much information about these terms, I will suggest that you should research about these topics. For variable selection, I follow following steps :

  1. Brainstorming : This involves your understanding of the problem you are trying to solve. I will see that which variables are expected to impact my dependent variable. In this step, I include all variables for which I have slight business justification that they may impact my dependent variable.
  2. Correlation Matrix : Assuming that you selected only quantitative variables in previous step. I will calculate correlation matrix of dependent variable and all independent variables.
  3. Here I will have detail look at the correlation matrix. To start with, I will select few independent variables (I prefer to start with 2 or 3) which have higher correlation with dependent variables. And then I add variables with smaller correlations sequentially. While including variables in the model, I always keep multicollinearity in mind. In a sense, you can say that I follow a kind of stepwise variable addition. But I prefer to say it trial and error.
  4. While adding variables, I keep track of adjusted R-square and prediction accuracy on test data. At the point when adjusted R-Square starts decreasing, it means that you are overfitting the model. If I dont get acceptable level of prediction accuracy on test data then I reconsider all of above steps. An obvious remedy is to modify set of independent variables in your model (add new or remove existing). In some cases, you may also need to think that whether linear regression is a suitable model for your problem or not.
  5. Calculating prediction accuracy itself will attract different opinions. But to start with, you can try simple percentage error calculation.
  6. train() function from caret package may help you in reducing manual steps in trial and error.
  • $\begingroup$ Step 3 would seem to have all the deficiencies and problems of forward stepwise regression and, in addition, would be subjective. In step 4, if you're repeatedly evaluating prediction accuracy on test data, then those are no longer functioning as test data. All in all, this looks like a set of ad hoc approaches that could be considerably improved by using more disciplined model-fitting methods. $\endgroup$ – whuber Oct 7 '15 at 18:05
  • $\begingroup$ Thank you all for the help. Have an idea of what to do now! $\endgroup$ – mapiye Oct 9 '15 at 8:18

I generally agree with user3697157.

Assuming your variables are continuous and not categorical, I would start with an x-y plot matrix (near the bottom of this page http://heather.cs.ucdavis.edu/DataAnalysisR/Lesson3.html) and a corrplot (https://cran.r-project.org/web/packages/corrplot/vignettes/corrplot-intro.html) to find good predictors.

Correlations and R^2 are related (http://mathbits.com/MathBits/TISection/Statistics2/correlation.htm).

You can also use VIFs to eliminate co-linear variables (see here https://onlinecourses.science.psu.edu/stat501/node/347). There is a vif function in the car package for R.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.