Choosing variables to include in a multiple linear regression model I am currently working to build a model using a multiple linear regression. After fiddling around with my model, I am unsure how to best determine which variables to keep and which to remove.
My model started with 10 predictors for the DV. When using all 10 predictors, four were considered significant. If I remove only some of the obviously-incorrect predictors, some of my predictors that were not initially significant become significant. Which leads me to my question: How does one go about determining which predictors to include in their model? It seemed to me you should run the model once with all predictors, remove those that are not significant, and then rerun. But if removing only some of those predictors makes others significant, I am left wondering if I am taking the wrong approach to all this.
I believe that this thread is similar to my question, but I am unsure I am interpreting the discussion correctly. Perhaps this is more of an experimental design topic, but maybe someone has some experience they can share.
 A: Based on your reaction to my comment:
You are looking for prediction. Thus, you should not really rely on (in)significance of the coefficients. You would be better to


*

*Pick a criterion that describes your prediction needs best (e.g.
missclassification rate, AUC of ROC, some form of these with
weights,...)

*For each model of interest, evaluate this criterion. This can be
done e.g.by providing a validation set (if you're lucky or rich),
through crossvalidation (typically tenfold), or whatever other
options your criterion of interest allows. If possible also find an
estimate of the SE of the criterion for each model (e.g. by using the
values over the different folds in crossvalidation)

*Now you can pick the model with the best value of the criterion,
though it is typically advised to pick the most parsimoneous model
(least variables) that is within one SE of the best value.


Wrt each model of interest: herein lies quite a catch. With 10 potential predictors, that is a truckload of potential models. If you've got the time or the processors for this (or if your data is small enough so that models get fit and evaluated fast enough): have a ball. If not, you can go about this by educated guesses, forward or backward modelling (but using the criterion instead of significance), or better yet: use some algorithm that picks a reasonable set of models. One algorithm that does this, is penalized regression, in particular Lasso regression. If you're using R, just plug in the package glmnet and you're about ready to go.
A: There is no simple answer to this.  When you remove some of the non-significant explanatory variables, others that are correlated with those may become significant.  There is nothing wrong with this, but it makes model selection at least partly art rather than science.  This is why experiments aim for keeping explanatory variables orthogonal to eachother, to avoid this problem.
Traditionally analysts did stepwise adding and subtracting of variables to the model one at a time (similar to what you have done) and testing them individually or in small groups with t or F tests.  The problem with this is you may miss some combination of variables to subract (or add) where their combined effect (or non-effect) is hidden by the collinearity.
With modern computing power it is feasible to fit all 2^10 = 1024 possible combinations of explanatory variables and choose the best model by one of a number of possible criteria eg AIC, BIC, or predictive power (for example, ability to predict the values of a test subset of the data that you have separated from the set you use to fit your model).  However, if you are going to be testing (implicitly or explicitly) 1024 models you will need to rethink your p-values from the classical approach - treat with caution...
A: If you are only interested in predictive performance, then it is probably better to use all of the features and use ridge-regression to avoid over-fitting the training sample.  This is essentially the advice given in the appendix of Millar's monograph on "subset selection in regression", so it comes with a reasonable pedigree!
The reason for this is that if you choose a subset based on a performance estimate based on a fixed sample of data (e.g. AIC, BIC, cross-validation etc.), the selection criterion will have a finite variance and so it is possible to over-fit the selection criterion itself.  In other words, to begin with as you minimise the selection criterion, generalisation performance will improve, however there will come a point where the more you reduce the selection criterion, the worse generalisation becomes.  If you are unlucky, you can easily end up with a regression model that performs worse than the one you started with (i.e. a model with all of the attributes).
This is especially likely when the dataset is small (so the selection criterion has a high variance) and when there are many possible choices of model (e.g. choosing combinations of features).  Regularisation seems to be less prone to over-fitting as it is a scalar parameter that needs to be tuned and this gives a more constrained view of the complexity of the model, i.e. fewer effective degrees of freedom with which to over-fit the selection criterion.
A: You can also use the step function in the Akaike information criterion. Example below. https://en.wikipedia.org/wiki/Akaike_information_criterion
StepModel = step(ClimateChangeModel)

A: Use the leaps library. When you plot the variables the y-axis shows R^2 adjusted. You look at where the boxes are black at the highest R^2. This will show the variables you should use for your multiple linear regression. 
Wine example below:
library(leaps)
regsubsets.out <-
  regsubsets(Price ~ Year + WinterRain + AGST + HarvestRain + Age + FrancePop,
         data = wine,
         nbest = 1,       # 1 best model for each number of predictors
         nvmax = NULL,    # NULL for no limit on number of variables
         force.in = NULL, force.out = NULL,
         method = "exhaustive")
regsubsets.out

#----When you plot wherever R^2 is the highest with black boxes,
#so in our case AGST + HarvestRain + WinterRain + Age and the dependent var.is Price----#
summary.out <- summary(regsubsets.out)
as.data.frame(summary.out$outmat)
plot(regsubsets.out, scale = "adjr2", main = "Adjusted R^2")

A: My advisor offered another possible way to go about this. Run all of your variables once, and then remove those that fail to meet some threshold (we set our threshold as p < .25). Continue iterating that way until all variables fall below that .25 value, then report those values which are significant.
A: Why not doing correlation analysis First and then onclude in regression only those that corelate with Dv?
