I have data with five predictor variables.I want to determine which of the first five variables affect my response variable by constructing some kind of a statistical model in R. Now, i have checked for the assumptions and managed to fit (full) multiple linear model. Now, i am not so sure of how i go about the process of variable selection in R and which best method to use and how.
I think your queation does not has one answer. I am sure that different person will follow different approach. In my opinion, variable selection is more of art and this is where experience plays vital role.
For variables selection/inclusion in regression model, I keep track of these three things - overfitting, multicollinearity, and prediction accuracy on test data. If you dont have much information about these terms, I will suggest that you should research about these topics. For variable selection, I follow following steps :
- Brainstorming : This involves your understanding of the problem you are trying to solve. I will see that which variables are expected to impact my dependent variable. In this step, I include all variables for which I have slight business justification that they may impact my dependent variable.
- Correlation Matrix : Assuming that you selected only quantitative variables in previous step. I will calculate correlation matrix of dependent variable and all independent variables.
- Here I will have detail look at the correlation matrix. To start with, I will select few independent variables (I prefer to start with 2 or 3) which have higher correlation with dependent variables. And then I add variables with smaller correlations sequentially. While including variables in the model, I always keep multicollinearity in mind. In a sense, you can say that I follow a kind of stepwise variable addition. But I prefer to say it trial and error.
- While adding variables, I keep track of adjusted R-square and prediction accuracy on test data. At the point when adjusted R-Square starts decreasing, it means that you are overfitting the model. If I dont get acceptable level of prediction accuracy on test data then I reconsider all of above steps. An obvious remedy is to modify set of independent variables in your model (add new or remove existing). In some cases, you may also need to think that whether linear regression is a suitable model for your problem or not.
- Calculating prediction accuracy itself will attract different opinions. But to start with, you can try simple percentage error calculation.
caretpackage may help you in reducing manual steps in trial and error.
I generally agree with user3697157.
Assuming your variables are continuous and not categorical, I would start with an x-y plot matrix (near the bottom of this page http://heather.cs.ucdavis.edu/DataAnalysisR/Lesson3.html) and a corrplot (https://cran.r-project.org/web/packages/corrplot/vignettes/corrplot-intro.html) to find good predictors.
Correlations and R^2 are related (http://mathbits.com/MathBits/TISection/Statistics2/correlation.htm).
You can also use VIFs to eliminate co-linear variables (see here https://onlinecourses.science.psu.edu/stat501/node/347). There is a vif function in the car package for R.