51
$\begingroup$

I am currently working to build a model using a multiple linear regression. After fiddling around with my model, I am unsure how to best determine which variables to keep and which to remove.

My model started with 10 predictors for the DV. When using all 10 predictors, four were considered significant. If I remove only some of the obviously-incorrect predictors, some of my predictors that were not initially significant become significant. Which leads me to my question: How does one go about determining which predictors to include in their model? It seemed to me you should run the model once with all predictors, remove those that are not significant, and then rerun. But if removing only some of those predictors makes others significant, I am left wondering if I am taking the wrong approach to all this.

I believe that this thread is similar to my question, but I am unsure I am interpreting the discussion correctly. Perhaps this is more of an experimental design topic, but maybe someone has some experience they can share.

$\endgroup$
5
  • 1
    $\begingroup$ The answer to this depends highly upon your goals and requirements: are you looking for simple association, or are you aiming for prediction; how high are you on interpretability; do you have any information on the variables from other publications that could influence the process; how about interactions or tranformed versions of the variables: can you include those; etc. You need to specify more details on what you're trying to do to get a good answer. $\endgroup$
    – Nick Sabbe
    Jan 18, 2012 at 8:40
  • $\begingroup$ Based on what you asked, this will be for prediction. Influence on other variables just offers possible association. There are no interactions between them. Only one value needs to be transformed, and it has been done. $\endgroup$ Jan 18, 2012 at 9:33
  • 1
    $\begingroup$ Is there a theory that says what predictors you should include? If you have a lot of variables that you have measured, and no theory, I would recommend holding out a set of observations so you can test your model on data that was not used to create it. It is not correct to test and validate a model on the same data. $\endgroup$
    – Michelle
    Jan 25, 2012 at 23:39
  • 1
    $\begingroup$ Cross validation (as Nick Sabbe discusses), penalized methods (Dikran Marsupial), or choosing variables based on prior theory (Michelle) are all options. But note that variable selection is intrinsically a very difficult task. To understand why it is so potentially fraught, it may help to read my answer here: algorithms-for-automatic-model-selection. Lastly, it's worth recognizing the problem is w/ the logical structure of this activity, not whether the computer does it for you automatically, or you do it manually for yourself. $\endgroup$ Nov 24, 2012 at 18:21
  • $\begingroup$ Check also out answers to this post: stats.stackexchange.com/questions/34769/… $\endgroup$
    – jokel
    Dec 8, 2012 at 18:24

6 Answers 6

24
$\begingroup$

Based on your reaction to my comment:

You are looking for prediction. Thus, you should not really rely on (in)significance of the coefficients. You would be better to

  • Pick a criterion that describes your prediction needs best (e.g. missclassification rate, AUC of ROC, some form of these with weights,...)
  • For each model of interest, evaluate this criterion. This can be done e.g.by providing a validation set (if you're lucky or rich), through crossvalidation (typically tenfold), or whatever other options your criterion of interest allows. If possible also find an estimate of the SE of the criterion for each model (e.g. by using the values over the different folds in crossvalidation)
  • Now you can pick the model with the best value of the criterion, though it is typically advised to pick the most parsimoneous model (least variables) that is within one SE of the best value.

Wrt each model of interest: herein lies quite a catch. With 10 potential predictors, that is a truckload of potential models. If you've got the time or the processors for this (or if your data is small enough so that models get fit and evaluated fast enough): have a ball. If not, you can go about this by educated guesses, forward or backward modelling (but using the criterion instead of significance), or better yet: use some algorithm that picks a reasonable set of models. One algorithm that does this, is penalized regression, in particular Lasso regression. If you're using R, just plug in the package glmnet and you're about ready to go.

$\endgroup$
2
  • $\begingroup$ +1, but could you explain why exactly you would "pick the most parsimoneous model (least variables) that is within one SE of the best value"? $\endgroup$
    – rolando2
    Jan 25, 2012 at 23:22
  • $\begingroup$ Parsimony is, for most situations, a wanted property: it heightens interpretability, and reduces the number of measurements you need to make for a new subject to use the model. The other side of the story is that what you get for your criterion is but an estimate, with matching SE: I've seen quite a few plots showing the criterion estimates against some tuning parameter, where the 'best' value was just an exceptional peak. As such, the 1 SE-rule (which is arbitrary, but an accepted practice) protects you from this with the added value of providing more parsimony. $\endgroup$
    – Nick Sabbe
    Jan 26, 2012 at 10:07
21
$\begingroup$

There is no simple answer to this. When you remove some of the non-significant explanatory variables, others that are correlated with those may become significant. There is nothing wrong with this, but it makes model selection at least partly art rather than science. This is why experiments aim for keeping explanatory variables orthogonal to eachother, to avoid this problem.

Traditionally analysts did stepwise adding and subtracting of variables to the model one at a time (similar to what you have done) and testing them individually or in small groups with t or F tests. The problem with this is you may miss some combination of variables to subract (or add) where their combined effect (or non-effect) is hidden by the collinearity.

With modern computing power it is feasible to fit all 2^10 = 1024 possible combinations of explanatory variables and choose the best model by one of a number of possible criteria eg AIC, BIC, or predictive power (for example, ability to predict the values of a test subset of the data that you have separated from the set you use to fit your model). However, if you are going to be testing (implicitly or explicitly) 1024 models you will need to rethink your p-values from the classical approach - treat with caution...

$\endgroup$
2
  • $\begingroup$ Thanks for the high level walk through of the pluses and minuses of both sides. It confirmed a lot of what I suspected. $\endgroup$ Jan 18, 2012 at 23:30
  • $\begingroup$ Wouldn't PCA be a good choice here? (You mentioned orthogonality) $\endgroup$
    – 24n8
    Jul 9, 2020 at 23:55
17
$\begingroup$

If you are only interested in predictive performance, then it is probably better to use all of the features and use ridge-regression to avoid over-fitting the training sample. This is essentially the advice given in the appendix of Millar's monograph on "subset selection in regression", so it comes with a reasonable pedigree!

The reason for this is that if you choose a subset based on a performance estimate based on a fixed sample of data (e.g. AIC, BIC, cross-validation etc.), the selection criterion will have a finite variance and so it is possible to over-fit the selection criterion itself. In other words, to begin with as you minimise the selection criterion, generalisation performance will improve, however there will come a point where the more you reduce the selection criterion, the worse generalisation becomes. If you are unlucky, you can easily end up with a regression model that performs worse than the one you started with (i.e. a model with all of the attributes).

This is especially likely when the dataset is small (so the selection criterion has a high variance) and when there are many possible choices of model (e.g. choosing combinations of features). Regularisation seems to be less prone to over-fitting as it is a scalar parameter that needs to be tuned and this gives a more constrained view of the complexity of the model, i.e. fewer effective degrees of freedom with which to over-fit the selection criterion.

$\endgroup$
1
$\begingroup$

You can also use the step function in the Akaike information criterion. Example below. https://en.wikipedia.org/wiki/Akaike_information_criterion

StepModel = step(ClimateChangeModel)
$\endgroup$
2
0
$\begingroup$

Use the leaps library. When you plot the variables the y-axis shows R^2 adjusted. You look at where the boxes are black at the highest R^2. This will show the variables you should use for your multiple linear regression.

Wine example below:

library(leaps)
regsubsets.out <-
  regsubsets(Price ~ Year + WinterRain + AGST + HarvestRain + Age + FrancePop,
         data = wine,
         nbest = 1,       # 1 best model for each number of predictors
         nvmax = NULL,    # NULL for no limit on number of variables
         force.in = NULL, force.out = NULL,
         method = "exhaustive")
regsubsets.out

#----When you plot wherever R^2 is the highest with black boxes,
#so in our case AGST + HarvestRain + WinterRain + Age and the dependent var.is Price----#
summary.out <- summary(regsubsets.out)
as.data.frame(summary.out$outmat)
plot(regsubsets.out, scale = "adjr2", main = "Adjusted R^2")
$\endgroup$
3
  • $\begingroup$ This doesn't sound very distinct from so-called 'best subsets' selection, which has known problems. $\endgroup$ May 8, 2015 at 17:43
  • $\begingroup$ leaps explicitly computes the 'best subsets', although it doesn't advise you how to select among subsets of different size. (That being a matter between you and your statistical clergy.) $\endgroup$ Jun 27, 2017 at 19:06
  • $\begingroup$ Funny enough, leaps is based on "FORTRAN77 code by Alan Miller [...] which is described in more detail in his book 'Subset Selection in Regression'", a book which is mentioned by Dikran in another answer to this question :-) $\endgroup$
    – gosuto
    Sep 18, 2019 at 7:05
-3
$\begingroup$

Why not doing correlation analysis First and then onclude in regression only those that corelate with Dv?

$\endgroup$
3
  • 3
    $\begingroup$ This is generally a poor way of choosing which variables to select, see e.g. Is using correlation matrix to select predictors for regression correct? A correlation analysis is quite different to multiple regression, because in the latter case we need to think about "partialling out" (regression slopes show the relationship once other variables are taken into account), but a correlation matrix doesn't show this. $\endgroup$
    – Silverfish
    Jul 7, 2016 at 20:48
  • 1
    $\begingroup$ This does not provide an answer to the question. Once you have sufficient reputation you will be able to comment on any post; instead, provide answers that don't require clarification from the asker. - From Review $\endgroup$
    – Sycorax
    Jul 7, 2016 at 21:24
  • 3
    $\begingroup$ @GeneralAbrial It strikes me that this is an answer to the question, albeit a brief one. It isn't a good solution to the problem, but that's what up/downvotes are for. (I think the "why not" is intended as a rhetorical question, rather than a request for clarification from the author.) $\endgroup$
    – Silverfish
    Jul 7, 2016 at 21:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.