Why is variable selection necessary? Common data-based variable selection procedures (for example, forward, backward, stepwise, all subsets) tend to yield models with undesirable properties, including:


*

*Coefficients biased away from zero.

*Standard errors that are too small and confidence intervals that are
too narrow.

*Test statistics and p-values that do not have the advertised meaning.

*Estimates of model fit which are overly optimistic.

*Included terms which can be meaningless (e.g., exclusion of lower-order
terms).


Yet, variable selection procedures persist.  Given the problems with variable selection, why are these procedures necessary?  What motivates their use?
Some proposals to start the discussion....


*

*The desire for interpretable regression coefficients? (Misguided in a
model with many IVs?)

*Eliminate variance introduced by irrelevant variables?

*Eliminate unnecessary covariance/redundancies among the independent variables?

*Reduce the number of parameter estimates (issues of power, sample size)


Are there others?  Are the problems addressed by variable selection techniques more or less important than the problems variable selection procedures introduce?  When should they be used?  When should they not be used?
 A: Variable selection (without penalization) only makes things worse.  Variable selection has almost no chance of finding the "right" variables, and results in large overstatements of effects of remaining variables and huge understatement of standard errors.  It is a mistake to believe that variable selection done in the usual way helps one get around the "large p small n" problem.  The bottom line is the the final model is misleading in every way.  This is related to an astounding statement I read in an epidemiology paper: "We didn't have an adequate sample size to develop a multivariable model, so instead we performed all possible tests for 2x2 tables."
Any time the dataset at hand is used to eliminate variables, while making use of Y to make the decision, all statistical quantities will be distorted.  Typical variable selection is a mirage.
Edit: (Copying comments from below hidden by the fold)      
I don't want to be self-serving but my book Regression Modeling Strategies goes into this in some depth. Online materials including handouts may be found at my webpage. Some available methods are $L_2$ penalization (ridge regression), $L_1$ penalization (lasso), and the so-called elastic net (combination of $L_1$ and $L_2$). Or use data reduction (blinded to the response $Y$) before doing regression. My book spends more space on this than on penalization.
A: First of all, the disadvantages you mentioned are the effects of feature selection done wrong, i.e. overfitted, unfinished or overshoot. 
The "ideal" FS has two steps; first one is the removal of all variables unrelated to the DV (so called all relevant problem, very hard task, unrelated to the used model/classifier), the second is to limit the set to only those variables which can be optimally used by the model (for instance $e^Y$ and $Y$ are equally good in explaining $Y$, but the linear model will rather fail to use $e^Y$ in general case) -- this one is called minimal optimal. 
All relevant level gives an insight in what really drives the given process, so have explanatory value. Minimal optimal level (by design) gives as non-overfitted model working on as uncluttered data as possible.
Real-world FS just want to achieve one of those goals (usually the latter).
A: Variable selection is necessarily because most models don't deal well with a large number of irrelevant variables.  These variables will only introduce noise into your model, or worse, cause you to over-fit. It's a good idea to exclude these variables from analysis.
Furthermore, you can't include all the variables that exist in every analysis, because there's an infinite number of them out there.  At some point you have to draw the line, and it's good to do so in a rigorous manner.  Hence all the discussion on variable selection.
Most of the issues with variables selection can be dealt with by cross-validation, or by using a model with built-in penalization and feature selection (such as the elastic net for linear models).
If you're interested in some empirical results related to multiple variables causing over-fitting, check out the results of the Don't Overfit competition on Kaggle.
