20
$\begingroup$

Could anyone advise if the following makes sense:

I am dealing with an ordinary linear model with 4 predictors. I am in two minds whether to drop the least significant term. It's $p$-value is a little over 0.05. I have argued in favor of dropping it along these lines: Multiplying the estimate of this term by (for example) the interquartile range of the sample data for this variable, gives some meaning to the clinical effect that keeping this term has on the overall model. Since this number is very low, approximately equal to typical intra-day range of values that the variable can take when measuring it in a clinical setting, I see it as not clinically significant and could therefore be dropped to give a more parsimonious model, even though dropping it reduces the adjusted $R^2$ a little.

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ why do you seek a more parsimonius model? $\endgroup$
    – Michael Bishop
    Commented Oct 27, 2011 at 18:31
  • 3
    $\begingroup$ Isn't parsimony a good thing in itself ? The way I see it, a model with variables that add little or no explanatory power in a clinical sense, is worse than a smaller model without those variables, even if those variables are significant in a statistical sense $\endgroup$
    – LeelaSella
    Commented Oct 27, 2011 at 20:11
  • $\begingroup$ I decided to write an answer: stats.stackexchange.com/questions/17624/…. But in short, No, I don't think parsimony is a good thing in itself. It is sometimes useful for specific reasons. $\endgroup$
    – Michael Bishop
    Commented Oct 28, 2011 at 0:44
  • 2
    $\begingroup$ I agree with Michael. It is best to include variables with no apparent explanatory ability if they were given a chance to be "significant"; you've already spent those degrees of freedom. $\endgroup$
    – Frank Harrell
    Commented Oct 29, 2011 at 13:52
  • 1
    $\begingroup$ Keep in mind that predictors that are not significant regressors can still contribute non-zero amounts to the explained variance in the case of correlated regressors -- by influencing other significant regressors. Especially with only four predictors, if regressors are correlated, I'd argue in favor of keeping the non-significant one in the model. $\endgroup$
    – Torvon
    Commented Jan 30, 2013 at 11:26

6 Answers 6

Reset to default
20
$\begingroup$

I have never understood the wish for parsimony. Seeking parsimony destroys all aspects of statistical inference (bias of regression coefficients, standard errors, confidence intervals, P-values). A good reason to keep variables is that this preserves the accuracy of confidence intervals and other quantities. Think of it this way: there have only been developed two unbiased estimators of residual variance in ordinary multiple regression: (1) the estimate from the pre-specified (big) model, and (2) the estimate from a reduced model substituting generalized degrees of freedom (GDF) for apparent (reduced) regression degrees of freedom. GDF will be much closer to the number of candidate parameters than to the number of final "significant" parameters.

Here's another way to think of it. Suppose you were doing an ANOVA to compare 5 treatments, getting a 4 d.f. F-test. Then for some reason you look at pairwise differences between treatments using t-tests and decided to combine or remove some of the treatments (this is the same as doing stepwise selection using P, AIC, BIC, Cp on the 4 dummy variables). The resulting F-test with 1, 2, or 3 d.f. will have inflated type I error. The original F-test with 4 d.f. contained a perfect multiplicity adjustment.

Improve this answer
$\endgroup$
3
  • 3
    $\begingroup$ +1 Parsimony is something that often only makes sense in very specific contexts. There's no reason to be playing the bias vs. precision game if you have enough precision to do both. $\endgroup$
    – Fomite
    Commented Oct 30, 2011 at 2:01
  • 3
    $\begingroup$ +1 for a great answer. But what if you have multicollinearity and removing a variable reduces it? (This isn't the case in the original question, but often is in other data). Isn't the resulting model often superior in all sorts of ways (reduce variance of estimators, signs of coefficients more likely to reflect underlying theory, etc)? If you still use the correct (original model) degrees of freedom. $\endgroup$
    – Peter Ellis
    Commented Feb 13, 2012 at 23:08
  • 4
    $\begingroup$ It is still better to include both variables. The only price you pay is the increased standard error in estimating one of the variable's effects adjusted for the other one. Joint tests of the two collinear variables are very powerful as then they combine forces rather than compete against one another. Also if you want to delete a variable, the data are incapable of telling you which one to delete. $\endgroup$
    – Frank Harrell
    Commented Feb 13, 2012 at 23:30
20
$\begingroup$

These answers about selection of variables all assume that the cost of the observation of variables is 0.

And that is not true.

While the issue of selection of variables for a given model may or may not involve selection, the implications for future behavior DOES involve selection.

Consider the problem of predicting which college lineman will do best in the NFL. You are a scout. You must consider which qualities of the current linemen in the NFL are most predictive of their success. You measure 500 quantities, and begin the task of the selection of the quantities which will be needed in the future.

What should you do? Should you retain all 500? Should some (astrological sign, day of the week born on) be eliminated?

This is an important question, and is not academic. There is a cost to the observation of data, and the framework of cost-effectiveness suggests that some variables NEED NOT be observed in the future, since their value is low.

Improve this answer
$\endgroup$
1
  • 4
    $\begingroup$ +1: an important and interesting point. It also reveals that the question is incomplete, because it does not indicate the purpose of the model. (Costs would be less relevant for a scientific model that seeks to build an explanatory theory but would come to the fore in a predictive model intended for repeated use.) $\endgroup$
    – whuber ♦
    Commented Feb 2, 2012 at 15:20
8
$\begingroup$

The most common advice these days is to get the AIC of the two models and take the one with the lower AIC. So, if your full model has an AIC of -20 and the model without the weakest predictor has an AIC > -20 then you keep the full model. Some might argue that if the difference < 3 you keep the simpler one. I prefer the advice that you could use the BIC to break "ties" when the AIC's are within 3 of each other.

If you're using R then the command to get the AIC is... AIC.

I do have a textbook on modelling here from the early 90s suggesting that you drop all of your predictors that are not significant. However, this really means you'll drop independent of the complexity the predictor adds or subtracts from the model. It's also only for ANOVA where significance is about variability explained rather than the magnitude of the slope in light of what other things have been explained. The more modern advice of using AIC takes these factors into consideration. There's all kinds of reasons the non-significant predictor should be included even if it's not significant. For example, there may be correlation issues with other predictors to it may be a relatively simple predictor. If you want the simplest advice go with AIC and use BIC to break ties and use a difference of 3 as your window of equality. Otherwise, provide many many more details about the model and you could get more specific advice for your situation.

Improve this answer
$\endgroup$
10
  • $\begingroup$ Smaller is better in the R representation, yes? $\endgroup$
    – Aaron - mostly inactive
    Commented Oct 27, 2011 at 17:06
  • $\begingroup$ Thanks for your reply. I found that the difference in AIC between the two models is only 2. $\endgroup$
    – LeelaSella
    Commented Oct 27, 2011 at 17:08
  • $\begingroup$ The smaller model has a slightly larger AIC and BIC AIC: large-small AIC= -2 BIC: large-small BIC- 7.8 $\endgroup$
    – LeelaSella
    Commented Oct 27, 2011 at 17:20
  • $\begingroup$ Aaron.. oops... lower, fixed... $\endgroup$
    – John
    Commented Oct 27, 2011 at 20:45
  • 1
    $\begingroup$ Just to clear something up, this additional term is just another covariate, and there is very little collinearity. $\endgroup$
    – LeelaSella
    Commented Oct 29, 2011 at 18:42
7
$\begingroup$

There are at least two other possible reasons for keeping a variable: 1) It affects the parameters for OTHER variables. 2) The fact that it is small is clinically interesting in itself

To see about 1, you can look at the predicted values for each person from a model with and without the variable in the model. I suggest making a scatterplot of these two sets of values. If there are no big differences, then that's an argument against this reason

For 2, think about why you had this variable in the list of possible variables. Is it based on theory? Did other research find a large effect size?

Improve this answer
$\endgroup$
6
  • $\begingroup$ There is very little collinearity to speak of, so removing this variable makes very little difference to the others. That's an interesting point about it being clinically interesting if it were small. The data come from an exploratory survey where, at this stage at least, there is no reason to expect any one variable to be more significant than any other. However, there is intra-day fluctuation in this variable, so on the face of it, if an effect was similar in size to this fluctuation, it doesn't seem very clinically significant to me. $\endgroup$
    – LeelaSella
    Commented Oct 27, 2011 at 21:59
  • $\begingroup$ OK, then it sounds like a good candidate for removal. $\endgroup$
    – Peter Flom
    Commented Oct 28, 2011 at 10:03
  • $\begingroup$ @P Sellaz - if "the data come from an exploratory survey," does that mean participants selected themselves? I find @Frank Harrell's comments something to be reckoned with, but concern for the strict accuracy of p-values, confidence intervals, etc. becomes moot if the sample was self-selected. $\endgroup$
    – rolando2
    Commented Oct 29, 2011 at 20:23
  • $\begingroup$ I think it only becomes moot if you are not using them. $\endgroup$
    – Frank Harrell
    Commented Oct 30, 2011 at 15:04
  • $\begingroup$ @FrankHarrel - please clarify: "them" = ? $\endgroup$
    – rolando2
    Commented Feb 2, 2012 at 1:27
5
$\begingroup$

What are you using this model for? Is parsimony an important goal?

More parsimonious models are preferred in some situations, but I wouldn't say parsimony is a good thing in itself. Parsimonious models can be understood and communicated more easily, and parsimony can help guard against over-fitting, but often times these issues are not major concerns or can be addressed another way.

Approaching from the opposite direction, including an extra term in a regression equation has some benefits even in situations in which the extra term itself isn't of interest and it doesn't improve the model fit much... you may not think that it is an important variable to control for, but others might. Of course, there are other very important substantive reasons to exclude a variable, e.g. it might be caused by the outcome.

Improve this answer
$\endgroup$
4
$\begingroup$

From your wording it sounds as if you're inclined to drop the last predictor because its predictive value is low; a substantial change on that predictor would not imply a substantial change on the response variable. If that is the case, then i like this criterion for including/dropping the predictor. It's more grounded in practical reality than the AIC or BIC can be, and more explainable to your audience for this research.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Yes, that is precisely what I meant. $\endgroup$
    – LeelaSella
    Commented Oct 27, 2011 at 18:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.