35
$\begingroup$

I am currently working to build a model using a multiple linear regression. After fiddling around with my model, I am unsure how to best determine which variables to keep and which to remove.

My model started with 10 predictors for the DV. When using all 10 predictors, four were considered significant. If I remove only some of the obviously-incorrect predictors, some of my predictors that were not initially significant become significant. Which leads me to my question: How does one go about determining which predictors to include in their model? It seemed to me you should run the model once with all predictors, remove those that are not significant, and then rerun. But if removing only some of those predictors makes others significant, I am left wondering if I am taking the wrong approach to all this.

I believe that this thread is similar to my question, but I am unsure I am interpreting the discussion correctly. Perhaps this is more of an experimental design topic, but maybe someone has some experience they can share.

$\endgroup$
  • $\begingroup$ The answer to this depends highly upon your goals and requirements: are you looking for simple association, or are you aiming for prediction; how high are you on interpretability; do you have any information on the variables from other publications that could influence the process; how about interactions or tranformed versions of the variables: can you include those; etc. You need to specify more details on what you're trying to do to get a good answer. $\endgroup$ – Nick Sabbe Jan 18 '12 at 8:40
  • $\begingroup$ Based on what you asked, this will be for prediction. Influence on other variables just offers possible association. There are no interactions between them. Only one value needs to be transformed, and it has been done. $\endgroup$ – cryptic_star Jan 18 '12 at 9:33
  • 1
    $\begingroup$ Is there a theory that says what predictors you should include? If you have a lot of variables that you have measured, and no theory, I would recommend holding out a set of observations so you can test your model on data that was not used to create it. It is not correct to test and validate a model on the same data. $\endgroup$ – Michelle Jan 25 '12 at 23:39
  • $\begingroup$ Cross validation (as Nick Sabbe discusses), penalized methods (Dikran Marsupial), or choosing variables based on prior theory (Michelle) are all options. But note that variable selection is intrinsically a very difficult task. To understand why it is so potentially fraught, it may help to read my answer here: algorithms-for-automatic-model-selection. Lastly, it's worth recognizing the problem is w/ the logical structure of this activity, not whether the computer does it for you automatically, or you do it manually for yourself. $\endgroup$ – gung Nov 24 '12 at 18:21
  • $\begingroup$ Check also out answers to this post: stats.stackexchange.com/questions/34769/… $\endgroup$ – jokel Dec 8 '12 at 18:24
19
$\begingroup$

Based on your reaction to my comment:

You are looking for prediction. Thus, you should not really rely on (in)significance of the coefficients. You would be better to

  • Pick a criterion that describes your prediction needs best (e.g. missclassification rate, AUC of ROC, some form of these with weights,...)
  • For each model of interest, evaluate this criterion. This can be done e.g.by providing a validation set (if you're lucky or rich), through crossvalidation (typically tenfold), or whatever other options your criterion of interest allows. If possible also find an estimate of the SE of the criterion for each model (e.g. by using the values over the different folds in crossvalidation)
  • Now you can pick the model with the best value of the criterion, though it is typically advised to pick the most parsimoneous model (least variables) that is within one SE of the best value.

Wrt each model of interest: herein lies quite a catch. With 10 potential predictors, that is a truckload of potential models. If you've got the time or the processors for this (or if your data is small enough so that models get fit and evaluated fast enough): have a ball. If not, you can go about this by educated guesses, forward or backward modelling (but using the criterion instead of significance), or better yet: use some algorithm that picks a reasonable set of models. One algorithm that does this, is penalized regression, in particular Lasso regression. If you're using R, just plug in the package glmnet and you're about ready to go.

$\endgroup$
  • $\begingroup$ +1, but could you explain why exactly you would "pick the most parsimoneous model (least variables) that is within one SE of the best value"? $\endgroup$ – rolando2 Jan 25 '12 at 23:22
  • $\begingroup$ Parsimony is, for most situations, a wanted property: it heightens interpretability, and reduces the number of measurements you need to make for a new subject to use the model. The other side of the story is that what you get for your criterion is but an estimate, with matching SE: I've seen quite a few plots showing the criterion estimates against some tuning parameter, where the 'best' value was just an exceptional peak. As such, the 1 SE-rule (which is arbitrary, but an accepted practice) protects you from this with the added value of providing more parsimony. $\endgroup$ – Nick Sabbe Jan 26 '12 at 10:07
13
$\begingroup$

There is no simple answer to this. When you remove some of the non-significant explanatory variables, others that are correlated with those may become significant. There is nothing wrong with this, but it makes model selection at least partly art rather than science. This is why experiments aim for keeping explanatory variables orthogonal to eachother, to avoid this problem.

Traditionally analysts did stepwise adding and subtracting of variables to the model one at a time (similar to what you have done) and testing them individually or in small groups with t or F tests. The problem with this is you may miss some combination of variables to subract (or add) where their combined effect (or non-effect) is hidden by the collinearity.

With modern computing power it is feasible to fit all 2^10 = 1024 possible combinations of explanatory variables and choose the best model by one of a number of possible criteria eg AIC, BIC, or predictive power (for example, ability to predict the values of a test subset of the data that you have separated from the set you use to fit your model). However, if you are going to be testing (implicitly or explicitly) 1024 models you will need to rethink your p-values from the classical approach - treat with caution...

$\endgroup$
  • $\begingroup$ Thanks for the high level walk through of the pluses and minuses of both sides. It confirmed a lot of what I suspected. $\endgroup$ – cryptic_star Jan 18 '12 at 23:30
11
$\begingroup$

If you are only interested in predictive performance, then it is probably better to use all of the features and use ridge-regression to avoid over-fitting the training sample. This is essentially the advice given in the appendix of Millar's monograph on "subset selection in regression", so it comes with a reasonable pedigree!

The reason for this is that if you choose a subset based on a performance estimate based on a fixed sample of data (e.g. AIC, BIC, cross-validation etc.), the selection criterion will have a finite variance and so it is possible to over-fit the selection criterion itself. In other words, to begin with as you minimise the selection criterion, generalisation performance will improve, however there will come a point where the more you reduce the selection criterion, the worse generalisation becomes. If you are unlucky, you can easily end up with a regression model that performs worse than the one you started with (i.e. a model with all of the attributes).

This is especially likely when the dataset is small (so the selection criterion has a high variance) and when there are many possible choices of model (e.g. choosing combinations of features). Regularisation seems to be less prone to over-fitting as it is a scalar parameter that needs to be tuned and this gives a more constrained view of the complexity of the model, i.e. fewer effective degrees of freedom with which to over-fit the selection criterion.

$\endgroup$
0
$\begingroup$

Use the leaps library. When you plot the variables the y-axis shows R^2 adjusted. You look at where the boxes are black at the highest R^2. This will show the variables you should use for your multiple linear regression.

Wine example below:

library(leaps)
regsubsets.out <-
  regsubsets(Price ~ Year + WinterRain + AGST + HarvestRain + Age + FrancePop,
         data = wine,
         nbest = 1,       # 1 best model for each number of predictors
         nvmax = NULL,    # NULL for no limit on number of variables
         force.in = NULL, force.out = NULL,
         method = "exhaustive")
regsubsets.out

#----When you plot wherever R^2 is the highest with black boxes,
#so in our case AGST + HarvestRain + WinterRain + Age and the dependent var.is Price----#
summary.out <- summary(regsubsets.out)
as.data.frame(summary.out$outmat)
plot(regsubsets.out, scale = "adjr2", main = "Adjusted R^2")
$\endgroup$
  • $\begingroup$ This doesn't sound very distinct from so-called 'best subsets' selection, which has known problems. $\endgroup$ – gung May 8 '15 at 17:43
  • $\begingroup$ leaps explicitly computes the 'best subsets', although it doesn't advise you how to select among subsets of different size. (That being a matter between you and your statistical clergy.) $\endgroup$ – steveo'america Jun 27 '17 at 19:06
  • $\begingroup$ Funny enough, leaps is based on "FORTRAN77 code by Alan Miller [...] which is described in more detail in his book 'Subset Selection in Regression'", a book which is mentioned by Dikran in another answer to this question :-) $\endgroup$ – jorijnsmit Sep 18 at 7:05
0
$\begingroup$

You can also use the step function in the Akaike information criterion. Example below. https://en.wikipedia.org/wiki/Akaike_information_criterion

StepModel = step(ClimateChangeModel)
$\endgroup$
  • $\begingroup$ Stepwise selection methods have lots of well-known problems. This isn't a good suggestion. $\endgroup$ – gung May 9 '15 at 16:42
  • $\begingroup$ @consent with gung some problems are discussed at stats.stackexchange.com/questions/69452/… $\endgroup$ – SIslam Mar 16 '17 at 12:04
-2
$\begingroup$

Why not doing correlation analysis First and then onclude in regression only those that corelate with Dv?

$\endgroup$
  • 2
    $\begingroup$ This is generally a poor way of choosing which variables to select, see e.g. Is using correlation matrix to select predictors for regression correct? A correlation analysis is quite different to multiple regression, because in the latter case we need to think about "partialling out" (regression slopes show the relationship once other variables are taken into account), but a correlation matrix doesn't show this. $\endgroup$ – Silverfish Jul 7 '16 at 20:48
  • $\begingroup$ This does not provide an answer to the question. Once you have sufficient reputation you will be able to comment on any post; instead, provide answers that don't require clarification from the asker. - From Review $\endgroup$ – Sycorax Jul 7 '16 at 21:24
  • 1
    $\begingroup$ @GeneralAbrial It strikes me that this is an answer to the question, albeit a brief one. It isn't a good solution to the problem, but that's what up/downvotes are for. (I think the "why not" is intended as a rhetorical question, rather than a request for clarification from the author.) $\endgroup$ – Silverfish Jul 7 '16 at 21:40
-4
$\begingroup$

My advisor offered another possible way to go about this. Run all of your variables once, and then remove those that fail to meet some threshold (we set our threshold as p < .25). Continue iterating that way until all variables fall below that .25 value, then report those values which are significant.

$\endgroup$
  • 1
    $\begingroup$ Hi allie, that is what @Peter Ellis mentioned in the second paragraph of his answer. His second sentence there covers the problem that this technique introduces. Do you have a theory, which is telling you what predictors to put into your model? $\endgroup$ – Michelle Jan 25 '12 at 18:07
  • $\begingroup$ Yes, @Michelle is right to underscore the liability to this approach. It can produce very arbitrary results. $\endgroup$ – rolando2 Jan 25 '12 at 23:18
  • $\begingroup$ Yes, there is a theory behind it all, which we are hoping to expand on. In particular, we are looking at how certain social cues (such as speech) interact. We are aware which ones do or don't already have clout. However, we are attempting to provide finer-grained versions. So, speech may be broken down into question, opinion, assessment, etc. $\endgroup$ – cryptic_star Jan 26 '12 at 1:29
  • 2
    $\begingroup$ Okay, so you're doing exploratory analysis. :) You can try different combinations, but you'll need to test the model you end up with on new data. By definition, with what you're doing, you'll have the "best" model for your data, but it may not work if you collect another set of data. $\endgroup$ – Michelle Jan 26 '12 at 9:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.