34
$\begingroup$

Common data-based variable selection procedures (for example, forward, backward, stepwise, all subsets) tend to yield models with undesirable properties, including:

  1. Coefficients biased away from zero.
  2. Standard errors that are too small and confidence intervals that are too narrow.
  3. Test statistics and p-values that do not have the advertised meaning.
  4. Estimates of model fit which are overly optimistic.
  5. Included terms which can be meaningless (e.g., exclusion of lower-order terms).

Yet, variable selection procedures persist. Given the problems with variable selection, why are these procedures necessary? What motivates their use?

Some proposals to start the discussion....

  • The desire for interpretable regression coefficients? (Misguided in a model with many IVs?)
  • Eliminate variance introduced by irrelevant variables?
  • Eliminate unnecessary covariance/redundancies among the independent variables?
  • Reduce the number of parameter estimates (issues of power, sample size)

Are there others? Are the problems addressed by variable selection techniques more or less important than the problems variable selection procedures introduce? When should they be used? When should they not be used?

Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ In my opinion, to discuss a problem clearly, we need to first specify it in some good way and then formulate it in an appropriate mathematical form so that we can have a framework under which the problem is clearly discussed. For the variable selection problem for, e.g., linear regression models. It seems reasonable to first fix a model, and study (i) the advantages/disadvantages (e.g., improvement/worsening in estimation or prediction) of variable selection? (ii) the advantages of variable selection procedure compared to the LS estimate? $\endgroup$
    – user87328
    Sep 1, 2015 at 8:48

3 Answers 3

Reset to default
21
$\begingroup$

Variable selection (without penalization) only makes things worse. Variable selection has almost no chance of finding the "right" variables, and results in large overstatements of effects of remaining variables and huge understatement of standard errors. It is a mistake to believe that variable selection done in the usual way helps one get around the "large p small n" problem. The bottom line is the the final model is misleading in every way. This is related to an astounding statement I read in an epidemiology paper: "We didn't have an adequate sample size to develop a multivariable model, so instead we performed all possible tests for 2x2 tables."

Any time the dataset at hand is used to eliminate variables, while making use of Y to make the decision, all statistical quantities will be distorted. Typical variable selection is a mirage.

Edit: (Copying comments from below hidden by the fold)

I don't want to be self-serving but my book Regression Modeling Strategies goes into this in some depth. Online materials including handouts may be found at my webpage. Some available methods are $L_2$ penalization (ridge regression), $L_1$ penalization (lasso), and the so-called elastic net (combination of $L_1$ and $L_2$). Or use data reduction (blinded to the response $Y$) before doing regression. My book spends more space on this than on penalization.

Improve this answer
$\endgroup$
11
  • 6
    $\begingroup$ I think this answer would be improved by providing some hints on how to proceed. The answer makes very broad and definitive statements (many of which I generally agree with) without reference to resources that would buttress the claims. Certainly penalization is not a panacea, either, and there are many choices to be made if one goes down that road. $\endgroup$
    – cardinal
    Nov 11, 2011 at 15:20
  • 3
    $\begingroup$ Please see above where I provided more information. The briefest way to state the problem is that a main reason that a variable is "selected" is because its effect was overestimated. $\endgroup$
    – Frank Harrell
    Nov 11, 2011 at 15:29
  • 2
    $\begingroup$ Yes, I agree your book has some good material on this, as does, for example, ESL. (That said, there are at least a couple of instances in ESL where some form of backward selection is also employed.) You mention $L_2$ penalization (aka ridge regression), but this generally does not get one too far in terms of variable/model selection per se. The elastic net has some ok behavior, but its drawback in my mind is that no matter how you look at it, it doesn't admit a very nice or natural "statistical" interpretation, whereas both $L_1$ and $L_2$ penalizations do in certain senses. $\endgroup$
    – cardinal
    Nov 11, 2011 at 15:35
  • 2
    $\begingroup$ Good points although I think that $L_{2}$ does give a natural interpretation because it's just another way of estimating the same model coefficients. You're right that $L_{2}$ without $L_{1}$ does not remove any variables. We do it for superior predictive performance and to handle the large $p$ small $n$ case. $\endgroup$
    – Frank Harrell
    Nov 11, 2011 at 15:50
  • 2
    $\begingroup$ Perhaps my comment was not quite as clear as I intended. Yes, I agree that $L_2$ penalization by itself has multiple nice interpretations, even though it does not result in any variable selection. It is the elastic net that I don't find particularly well-motivated or natural from a statistical perspective beyond the fact that in some cases better predictive performance is achieved. $\endgroup$
    – cardinal
    Nov 11, 2011 at 16:05
14
$\begingroup$

First of all, the disadvantages you mentioned are the effects of feature selection done wrong, i.e. overfitted, unfinished or overshoot.

The "ideal" FS has two steps; first one is the removal of all variables unrelated to the DV (so called all relevant problem, very hard task, unrelated to the used model/classifier), the second is to limit the set to only those variables which can be optimally used by the model (for instance $e^Y$ and $Y$ are equally good in explaining $Y$, but the linear model will rather fail to use $e^Y$ in general case) -- this one is called minimal optimal.

All relevant level gives an insight in what really drives the given process, so have explanatory value. Minimal optimal level (by design) gives as non-overfitted model working on as uncluttered data as possible.

Real-world FS just want to achieve one of those goals (usually the latter).

Improve this answer
$\endgroup$
6
  • 4
    $\begingroup$ I assume you are referring to removing variables without using the data at hand. You cannot use the dataset at hand to do this. This would be unreliable and distort statistical inference. $\endgroup$
    – Frank Harrell
    Nov 11, 2011 at 14:40
  • $\begingroup$ As I wrote, this is just a theoretical foundation of the problem (coming from Bayesian nets). The exact way of realizing this is obviously impossible, and I certainly agree that statistical modelling have suffered a lot from mindless use of RFE and similar stuff -- yet machine learning has some heuristic algorithms which certainly are not hopeless (i.e. make stable selections and models that prove to be not overfitted in fair tests). $\endgroup$
    – user88
    Nov 11, 2011 at 16:13
  • $\begingroup$ What is RFE??????? $\endgroup$
    – kjetil b halvorsen ♦
    Sep 1, 2015 at 9:03
  • $\begingroup$ @kjetilbhalvorsen Recursive Feature Elimination $\endgroup$
    – user88
    Sep 1, 2015 at 12:13
  • $\begingroup$ @mbq Thx for your interesting answer! Can you provide any reference(books, papers etc.)? Appreciate your reply! $\endgroup$
    – Carol.Kar
    Sep 13, 2015 at 10:35
10
$\begingroup$

Variable selection is necessarily because most models don't deal well with a large number of irrelevant variables. These variables will only introduce noise into your model, or worse, cause you to over-fit. It's a good idea to exclude these variables from analysis.

Furthermore, you can't include all the variables that exist in every analysis, because there's an infinite number of them out there. At some point you have to draw the line, and it's good to do so in a rigorous manner. Hence all the discussion on variable selection.

Most of the issues with variables selection can be dealt with by cross-validation, or by using a model with built-in penalization and feature selection (such as the elastic net for linear models).

If you're interested in some empirical results related to multiple variables causing over-fitting, check out the results of the Don't Overfit competition on Kaggle.

Improve this answer
$\endgroup$
3
  • 2
    $\begingroup$ I think the first paragraph contains a significant misunderstanding of the problem. Variable selection does not help with those problems in any way, it only hides them. Variable selection results in tremendous overfitting problems, although as you mentioned later there are some ways to honestly penalize ourselves for the damage caused by variable selection. $\endgroup$
    – Frank Harrell
    Nov 11, 2011 at 14:42
  • 3
    $\begingroup$ @Frank Harrell: how do you decide which variables to exclude from a model? $\endgroup$
    – Zach
    Nov 11, 2011 at 15:09
  • 12
    $\begingroup$ (1) Use subject matter knowledge before looking at the dataset; (2) Use redundancy analysis/data reduction blinded to Y; (3) Use a method that adequately penalizes for the huge multiple comparison problem caused by doing feature selection (see elsewhere on this page). $\endgroup$
    – Frank Harrell
    Nov 11, 2011 at 16:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.