Variable selection for predictive modeling really needed in 2016?

This question has been asked on CV some yrs ago, it seems worth a repost in light of 1) order of magnitude better computing technology (e.g. parallel computing, HPC etc) and 2) newer techniques, e.g. [3].

First, some context. Let's assume the goal is not hypothesis testing, not effect estimation, but prediction on un-seen test set. So, no weight is given to any interpretable benefit. Second, let's say you cannot rule out the relevance of any predictor on subject matter consideration, ie. they all seem plausible individually or in combination with other predictors. Third, you're confront with (hundreds of) millions of predictors. Fourth, let's say you have access to AWS with an unlimited budget, so computing power is not a constraint.

The usual reaons for variable selection are 1) efficiency; faster to fit a smaller model and cheaper to collect fewer predictors, 2) interpretation; knowing the "important" variables gives insight into the underlying process [1].

It's now widely known that many variable selection methods are ineffective and often outright dangerous (e.g. forward stepwise regression) [2].

Secondly, if the selected model is any good, one shouldn't need to cut down on the list of predictors at all. The model should do it for you. A good example is lasso, which assigns a zero coefficient to all the irrelevant variables.

I'm aware that some people advocate using an "elephant" model, ie. toss every conceivable predictors into the fit and run with it [2].

Is there any fundamental reason to do variable selection if the goal is predictive accuracy?

[1] Reunanen, J. (2003). Overfitting in making comparisons between variable selection methods. The Journal of Machine Learning Research, 3, 1371-1382.

[2] Harrell, F. (2015). Regression modeling strategies: with applications to linear models, logistic and ordinal regression, and survival analysis. Springer.

[3] Taylor, J., & Tibshirani, R. J. (2015). Statistical learning and selective inference. Proceedings of the National Academy of Sciences, 112(25), 7629-7634.

[4] Zhou, J., Foster, D., Stine, R., & Ungar, L. (2005, August). Streaming feature selection using alpha-investing. In Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining (pp. 384-393). ACM.

• Nice first question - it may possibly be closed as a duplicate, but I appreciate that you've expended effort setting out what you feel distinguishes it. I'd suggest editing the title, so it's clearer your focus is on prediction only. – Silverfish May 28 '16 at 20:22
• If this question was already asked but you find it important to re-post it after some time passed than maybe you could provide a link to the previous question? It could be interesting to be able to compare the previous answers. – Tim May 28 '16 at 20:42
• @qbert65536 One view is you don't. Feature selection is inherently unreliable. – horaceT May 28 '16 at 21:36
• Methods that automatically select a sparse subset of features (e.g. l1 penalized models) are also performing feature selection. So the critical question isn't "is feature selection good/bad", but "what are the properties the distinguish good feature selection methods from bad ones?". Being performed jointly with parameter estimation (as in lasso) is one property, and we could ask whether that matters (along with many other properties). – user20160 May 29 '16 at 4:28
• @ToussaintLouverture Since I posted this question a yr ago, I have second (and third) thought. Now I believe the appropriate question is, how important is it to direct effort toward variable selection, rather than to model selection to pick a more capable model that generalize from all the features of an experiment. – horaceT Oct 13 '17 at 16:03

There have been rumors for years that Google uses all available features in building its predictive algorithms. To date however, no disclaimers, explanations or white papers have emerged that clarify and/or dispute this rumor. Not even their published patents help in the understanding. As a result, no one external to Google knows what they are doing, to the best of my knowledge.

/* Update in Sept 2019, a Google Tensorflow evangelist went on record in a presentation in stating that Google engineers regularly evaluate over 5 billion parameters for the current version of PageRank. */

As the OP notes, one of the biggest problems in predictive modeling is the conflation between classic hypothesis testing and careful model specification vs pure data mining. The classically trained can get quite dogmatic about the need for "rigor" in model design and development. The fact is that when confronted with massive numbers of candidate predictors and multiple possible targets or dependent variables, the classic framework neither works, holds nor provides useful guidance. Numerous recent papers delineate this dilemma from Chattopadhyay and Lipson's brilliant paper Data Smashing: Uncovering Lurking Order in Data http://rsif.royalsocietypublishing.org/content/royinterface/11/101/20140826.full.pdf

The key bottleneck is that most data comparison algorithms today rely on a human expert to specify what ‘features’ of the data are relevant for comparison. Here, we propose a new principle for estimating the similarity between the sources of arbitrary data streams, using neither domain knowledge nor learning.

To last year's AER paper on Prediction Policy Problems by Kleinberg, et al.https://www.aeaweb.org/articles?id=10.1257/aer.p20151023 which makes the case for data mining and prediction as useful tools in economic policy making, citing instances where "causal inference is not central, or even necessary."

The fact is that the bigger, $64,000 question is the broad shift in thinking and challenges to the classic hypothesis-testing framework implicit in, e.g., this Edge.org symposium on "obsolete" scientific thinking https://www.edge.org/responses/what-scientific-idea-is-ready-for-retirement as well as this recent article by Eric Beinhocker on the "new economics" which presents some radical proposals for integrating widely different disciplines such as behavioral economics, complexity theory, predictive model development, network and portfolio theory as a platform for policy implementation and adoption https://evonomics.com/the-deep-and-profound-changes-in-economics-thinking/ Needless to say, these issues go far beyond merely economic concerns and suggest that we are undergoing a fundamental shift in scientific paradigms. The shifting views are as fundamental as the distinctions between reductionistic, Occam's Razor like model-building vs Epicurus' expansive Principle of Plenitude or multiple explanations which roughly states that if several findings explain something, retain them all ... https://en.wikipedia.org/wiki/Principle_of_plenitude Of course, guys like Beinhocker are totally unencumbered with practical, in the trenches concerns regarding applied, statistical solutions to this evolving paradigm. Wrt the nitty-gritty questions of ultra-high dimensional variable selection, the OP is relatively nonspecific regarding viable approaches to model building that might leverage, e.g., Lasso, LAR, stepwise algorithms or "elephant models” that use all of the available information. The reality is that, even with AWS or a supercomputer, you can't use all of the available information at the same time – there simply isn’t enough RAM to load it all in. What does this mean? Workarounds have been proposed, e.g., the NSF's Discovery in Complex or Massive Datasets: Common Statistical Themes to "divide and conquer" algorithms for massive data mining, e.g., Wang, et al's paper, A Survey of Statistical Methods and Computing for Big Data http://arxiv.org/pdf/1502.07989.pdf as well as Leskovec, et al's book Mining of Massive Datasets http://www.amazon.com/Mining-Massive-Datasets-Jure-Leskovec/dp/1107077230/ref=sr_1_1?ie=UTF8&qid=1464528800&sr=8-1&keywords=Mining+of+Massive+Datasets There are now literally hundreds, if not thousands of papers that deal with various aspects of these challenges, all proposing widely differing analytic engines as their core from the “divide and conquer” algorithms; unsupervised, "deep learning" models; random matrix theory applied to massive covariance construction; Bayesian tensor models to classic, supervised logistic regression, and more. Fifteen years or so ago, the debate largely focused on questions concerning the relative merits of hierarchical Bayesian solutions vs frequentist finite mixture models. In a paper addressing these issues, Ainslie, et al. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.197.788&rep=rep1&type=pdf came to the conclusion that the differing theoretical approaches, in practice, produced largely equivalent results with the exception of problems involving sparse and/or high dimensional data where HB models had the advantage. Today with the advent of D&C workarounds, any arbitrage HB models may have historically enjoyed are being eliminated. The basic logic of these D&C workarounds are, by and large, extensions of Breiman's famous random forest technique which relied on bootstrapped resampling of observations and features. Breiman did his work in the late 90s on a single CPU when massive data meant a few dozen gigs and a couple of thousand features. On today's massively parallel, multi-core platforms, it is possible to run algorithms analyzing terabytes of data containing tens of millions of features building millions of "RF" mini-models in a few hours. There are any number of important questions coming out of all of this. One has to do with a concern over a loss of precision due to the approximating nature of these workarounds. This issue has been addressed by Chen and Xie in their paper, A Split-and-Conquer Approach for Analysis of Extraordinarily Large Data http://dimacs.rutgers.edu/TechnicalReports/TechReports/2012/2012-01.pdf where they conclude that the approximations are indistinguishably different from the "full information" models. A second concern which, to the best of my knowledge hasn't been adequately addressed by the literature, has to do with what is done with the results (i.e., the "parameters") from potentially millions of predictive mini-models once the workarounds have been rolled up and summarized. In other words, how does one execute something as simple as "scoring" new data with these results? Are the mini-model coefficients to be saved and stored or does one simply rerun the d&c algorithm on new data? In his book, Numbers Rule Your World, Kaiser Fung describes the dilemma Netflix faced when presented with an ensemble of only 104 models handed over by the winners of their competition. The winners had, indeed, minimized the MSE vs all other competitors but this translated into only a several decimal place improvement in accuracy on the 5-point, Likert-type rating scale used by their movie recommender system. In addition, the IT maintenance required for this ensemble of models cost much more than any savings seen from the "improvement" in model accuracy. Then there's the whole question of whether "optimization" is even possible with information of this magnitude. For instance, Emmanuel Derman, the physicist and financial engineer, in his book My Life as a Quant suggests that optimization is an unsustainable myth, at least in financial engineering. Finally, important questions concerning relative feature importance with massive numbers of features have yet to be addressed. There are no easy answers wrt questions concerning the need for variable selection and the new challenges opened up by the current, Epicurean workarounds remain to be resolved. The bottom line is that we are all data scientists now. **** EDIT *** References 1. Chattopadhyay I, Lipson H. 2014 Data smashing: uncovering lurking order in data. J. R. Soc. Interface 11: 20140826. http://dx.doi.org/10.1098/rsif.2014.0826 2. Kleinberg, Jon, Jens Ludwig, Sendhil Mullainathan and Ziad Obermeyer. 2015. "Prediction Policy Problems." American Economic Review, 105(5): 491-95. DOI: 10.1257/aer.p20151023 3. Edge.org, 2014 Annual Question : WHAT SCIENTIFIC IDEA IS READY FOR RETIREMENT? https://www.edge.org/responses/what-scientific-idea-is-ready-for-retirement 4. Eric Beinhocker, How the Profound Changes in Economics Make Left Versus Right Debates Irrelevant, 2016, Evonomics.org. https://evonomics.com/the-deep-and-profound-changes-in-economics-thinking/ 5. Epicurus principle of multiple explanations: keep all models. Wikipedia https://www.coursehero.com/file/p6tt7ej/Epicurus-Principle-of-Multiple-Explanations-Keep-all-models-that-are-consistent/ 6. NSF, Discovery in Complex or Massive Datasets: Common Statistical Themes, A Workshop funded by the National Science Foundation, October 16-17, 2007 https://www.nsf.gov/mps/dms/documents/DiscoveryInComplexOrMassiveDatasets.pdf 7. Statistical Methods and Computing for Big Data, Working Paper by Chun Wang, Ming-Hui Chen, Elizabeth Schifano, Jing Wu, and Jun Yan, October 29, 2015 http://arxiv.org/pdf/1502.07989.pdf 8. Jure Leskovec, Anand Rajaraman, Jeffrey David Ullman, Mining of Massive Datasets, Cambridge University Press; 2 edition (December 29, 2014) ISBN: 978-1107077232 9. Large Sample Covariance Matrices and High-Dimensional Data Analysis (Cambridge Series in Statistical and Probabilistic Mathematics), by Jianfeng Yao, Shurong Zheng, Zhidong Bai, Cambridge University Press; 1 edition (March 30, 2015) ISBN: 978-1107065178 10. RICK L. ANDREWS, ANDREW AINSLIE, and IMRAN S. CURRIM, An Empirical Comparison of Logit Choice Models with Discrete Versus Continuous Representations of Heterogeneity, Journal of Marketing Research, 479 Vol. XXXIX (November 2002), 479–487 http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.197.788&rep=rep1&type=pdf 11. A Split-and-Conquer Approach for Analysis of Extraordinarily Large Data, Xueying Chen and Minge Xie, DIMACS Technical Report 2012-01, January 2012 http://dimacs.rutgers.edu/TechnicalReports/TechReports/2012/2012-01.pdf 12. Kaiser Fung, Numbers Rule Your World: The Hidden Influence of Probabilities and Statistics on Everything You Do, McGraw-Hill Education; 1 edition (February 15, 2010) ISBN: 978-0071626538 13. Emmanuel Derman, My Life as a Quant: Reflections on Physics and Finance, Wiley; 1 edition (January 11, 2016) ISBN: 978-0470192733 * Update in November 2017 * Nathan Kutz' 2013 book, Data-Driven Modeling & Scientific Computation: Methods for Complex Systems & Big Data is a mathematical and PDE-focused excursion into variable selection as well as dimension reduction methods and tools. An excellent, 1 hour introduction to his thinking can be found in this June 2017 Youtube video Data Driven Discovery of Dynamical Systems and PDEs . In it, he makes references to the latest developments in this field. https://www.youtube.com/watch?feature=youtu.be&v=Oifg9avnsH4&app=desktop • At the Machine Learning Summer School couple yrs ago, a fellow from Google gave a talk (forgot name). He mentioned a couple of (binary classification) models in production involve something like ~200 million features batch-trained on ~30 Tb of datasets; most of them are probably binary features. I don't remember he ever mentioned variable selection. – horaceT May 30 '16 at 18:32 • Great comments (though part of it went off on a tangent). I particularly like the perspective that many old-fashioned ideas need re-examination in the era of Big Data. – horaceT May 30 '16 at 18:50 • @horaceT Very interesting. At least that confirms the rumor. Thanks. Which ML program was that? – Mike Hunter May 30 '16 at 18:57 • MLSS 2012 at UC Santa Cruz. The speaker was Tushar Chandra, here is the slides, users.soe.ucsc.edu/~niejiazhong/slides/chandra.pdf – horaceT May 30 '16 at 18:58 • @Glen_b Thanks for the comments. I thought I provided names and titles for the references precisely because of the broken link issue. Regardless, I'll add a reference section at the end. Let me know if anything is missing. – Mike Hunter Jun 19 '16 at 13:08 In terms of prediction, you probably need to think of the question of how quickly the model learns the important features. Even thinking of OLS, this will give you something like model selection given enough data. But we know that it doesn't converge to this solution quickly enough - so we search for something better. Most methods are making an assumption about the kind of betas/coefficients that are going to be encountered (like a prior distribution in a bayesian model). They work best when these assumptions hold. For example, ridge/lasso regression assumes most betas are on the same scale with most near zero. They won't work as well for the "needles in a haystack" regressions where most betas are zero, and some betas are very large (i.e. scales are very different). Feature selection may work better here - lasso can get stuck in between shrinking noise and leaving signal untouched. Feature selection is more fickle - an effect is either "signal" or "noise". In terms of deciding - you need to have some idea of what sort of predictor variables you have. Do you have a few really good ones? Or all variables are weak? This will drive the profile of betas you will have. And which penalty/selection methods you use (horses for courses and all that). Feature selection is also not bad but some of the older approximations due to computational restrictions are no longer good (stepwise, forward). Model averaging using feature selection (all 1 var models, 2 var models, etc weighted by their performance) will do a pretty good job at prediction. But these are essentially penalising the betas through the weight given to models with that variable excluded - just not directly - and not in a convex optimisation problem sort of way. I give you the perspective of industry. Industries don't like to spend money on sensors and monitoring systems which they don't know how much they will benefit from. For instance, I don't want to name, so imagine a component with 10 sensors gathering data every minute. The asset owner turns to me and asks me how well can you predict the behavior of my component with these data from 10 sensors? Then they perform a cost-benefit analysis. Then, they have the same component with 20 sensors, they ask me, again, how well can you predict the behavior of my component with these data from 20 sensors? They perform another cost-benefit analysis. At each of these cases, they compare the benefit with the investment cost due to sensors installations. (This is not just adding a$10 sensor to a component. A lot of factors play a role). Here is where a variable selection analysis can be useful.

• Good point. But you wouldn't know 10 sensors good enough or need another 10 until you have some data from the 20. – horaceT May 29 '16 at 20:36
• True, and you can always speculate based on some studies. You install each sensor with a goal, to avoid failures. If the failure rates are low or you have already covered the significant parts of a component, you know addition of 1 sensor won't bring a large return. So, you don't need to install those sensors, collect data and perform a study to know whether those additional sensors are actually good enough. – PeyM87 May 30 '16 at 8:38
• 'Sensors' may not mean sensors - in my company, we subscribe to all our data, so there is indeed an opportunity to discover features which are not contributing to anything, and cut costs by removing them from the subscription service (to be clear, subscription rates are worked out at a higher level than individual columns, but certainly it is plausible to imagine an element of the subscription contributing one feature to a final model, and being able to discontinue if it doesn't improve performance) – Robert de Graaf Jun 20 '16 at 5:24

As part of an algorithm for learning a purely predictive model, variable selection is not necessarily bad from a performance viewpoint nor is it automatically dangerous. However, there are some issues that one should be aware of.

To make the question a little more concrete, let's consider the linear regression problem with $$E(Y_i \mid X_i) = X_i^T \beta$$ for $i = 1, \ldots, N$, and $X_i$ and $\beta$ being $p$-dimensional vectors of variables and parameters, respectively. The objective is to find a good approximation of the function $$x \mapsto E(Y \mid X = x) = X^T \beta,$$ which is the prediction of $Y$ given $X = x$. This can be achieved by estimating $\beta$ using combinations of variable selection and minimisation of a loss function with or without penalisation. Model averaging or Bayesian methods may also be used, but let's focus on single model predictions.

Stepwise selection algorithms like forward and backward variable selection can be seen as approximate attempts to solve a best subset selection problem, which is computationally hard (so hard that the improvements of computational power matters little). The interest is in finding for each $k = 1, \ldots, \min(N, p)$ the best (or at least a good) model with $k$ variables. Subsequently, we may optimise over $k$.

The danger with such a variable selection procedure is that many standard distributional results are invalid conditionally on the variable selection. This holds for standard tests and confidence intervals, and is one of the problems that Harrell [2] is warning about. Breiman also warned about model selection based on e.g. Mallows' $C_p$ in The Little Bootstrap .... Mallows' $C_p$, or AIC for that matter, do not account for the model selection, and they will give overly optimistic prediction errors.

However, cross-validation can be used for estimating the prediction error and for selecting $k$, and variable selection can achieve a good balance between bias and variance. This is particularly true if $\beta$ has a few large coordinates with the rest close to zero $-$ as @probabilityislogic mentions.

Shrinkage methods such as ridge regression and lasso can achieve a good tradeoff between bias and variance without explicit variable selection. However, as the OP mentions, lasso does implicit variable selection. It's not really the model but rather the method for fitting the model that does variable selection. From that perspective, variable selection (implicit or explicit) is simply part of the method for fitting the model to data, and it should be regarded as such.

Algorithms for computing the lasso estimator can benefit from variable selection (or screening). In Statistical Learning with Sparsity: The Lasso and Generalizations, Section 5.10, it it described how screening, as implemented in glmnet, is useful. It can lead to substantially faster computation of the lasso estimator.

One personal experience is from an example where variable selection made it possible to fit a more complicated model (a generalised additive model) using the selected variables. Cross-validation results indicated that this model was superior to a number of alternatives $-$ though not to a random forest. If gamsel had been around $-$ which integrates generalised additive models with variable selection $-$ I might have considered trying it out as well.

Edit: Since I wrote this answer there is a paper out on the particular application I had in mind. R-code for reproducing the results in the paper is available.

In summary I will say that variable selection (in one form or the other) is and will remain to be useful $-$ even for purely predictive purposes $-$ as a way to control the bias-variance tradeoff. If not for other reasons, then at least because more complicated models may not be able to handle very large numbers of variables out-of-the-box. However, as time goes we will naturally see developments like gamsel that integrate variable selection into the estimation methodology.

It is, of course, always essential that we regard variable selection as part of the estimation method. The danger is to believe that variable selection performs like an oracle and identifies the correct set of variables. If we believe that and proceed as if variables were not selected based on the data, then we are at risk of making errors.

• I'm not clear on how variable selection made it possible to fit a more complicated model. With variable selection you are still estimating the same large number of parameters; you're just estimating some of them as zero. Stability of a conditional model fitted after variable selection can be a mirage. – Frank Harrell May 29 '16 at 16:13
• @Harrell, in the particular example the variable selection was carried out using lasso in combination with stability selection in the model where all variables entered linearly. The gam was then fitted using the selected variables. I completely agree that variable selection is just estimating some parameters to zero, and the application did exactly that in a gam model by a two-step procedure. I'm sure that gamsel provides a more systematic approach. My point was that without such an approach, variable selection can be useful shortcut. – NRH May 29 '16 at 16:30
• Using an unpenalized method to re-fit variables selected in an earlier penalization phase is not appropriate. That would be substantially biased. And unpenalized variable selection is not a good shortcut. – Frank Harrell May 29 '16 at 17:35
• Stability selection is more conservative than selecting variables using lasso and re-fitting without penalisation. The latter did, expectedly, not work very well from a predictive viewpoint (as measure by cross-validation). When I via cross-validation in a concrete case find that variable selection + gam gives better predictive performance than the ridge or lasso estimator, then that is my measure of whether the procedure is good. – NRH May 29 '16 at 18:30
• Please define 'stability selection'. And re-fitting without penalization is anti-conservative. – Frank Harrell May 30 '16 at 13:06

Allow me to comment on the statement: “... fitting k parameters to n < k observations is just not going to happen.”

In chemometrics we are often interested in predictive models, and the situation k >> n is frequently encountered (e.g. in spectroscopic data). This problem is typically solved simply by projecting the observations to a lower dimensional subspace a, where a < n, before the regression (e.g. Principal Component Regression). Using Partial Least Squares Regression the projection and regression are performed simultaneously favoring quality of prediction. The methods mentioned find optimal pseudo-inverses to a (singular) covariance or correlation matrix, e.g. by singular value decomposition.

Experience shows that predictive performance of multivariate models increases when noisy variables are removed. So even if we - in a meaningful way - are able to estimate k parameters having only n equations (n < k), we strive for parsimonious models. For that purpose, variable selection becomes relevant, and much chemometric literature are devoted to this subject.

While prediction is an important objective, the projection methods at the same time offers valuable insight into e.g. patterns in data and relevance of variables. This is facilitated mainly by diverse model-plots, e.g. scores, loadings, residuals, etc...

Chemometric technology is used extensively e.g. in the industry where reliable and accurate predictions really count.

In several well known cases, yes, variable selection is not necessary. Deep learning has become a bit overhyped for precisely this reason.

For example, when a convoluted neural network (http://cs231n.github.io/convolutional-networks/) tries to predict if a centered image contains a human face, the corners of the image tend to have minimal predictive value. Traditional modeling and variable selection would have the modeler remove the corner pixels as predictors; however, the convoluted neural network is smart enough to essentially discard these predictors automatically. This is true for most deep learning models that try to predict the presence of some object in an image (e.g., self drivings cars "predicting" lane markings, obstacles or other cars in frames of onboard streaming video).

Deep learning is probably overkill for a lot of traditional problems such as where datasets are small or where domain knowledge is abundant, so traditional variable selection will probably remain relevant for a long time, at least in some areas. Nonetheless, deep learning is great when you want to throw together a "pretty good" solution with minimal human intervention. It might take me many hours to handcraft and select predictors to recognize handwritten digits in images, but with a convoluted neural network and zero variable selection, I can have a state-of-the-art model in just under 20 minutes using Google's TensorFlow (https://www.tensorflow.org/versions/r0.8/tutorials/mnist/pros/index.html).

• I really like this DL perspective. In Computer Vision, the data matrices you encounter are flattened 2D images, where the meaning of a particular column depends on the observation. Example, pixel 147 may be the face of a cat in image No. 27, but it's the background wall in image No. 42. So, feature selection as we know it would fail miserably. That's why ConvNet is so powerful because it has build-in translational/rotational invariance. – horaceT May 30 '16 at 18:22