For the selection of predictors in multivariate linear regression with $p$ suitable predictors, what methods are available to find an 'optimal' subset of the predictors without explicitly testing all $2^p$ subsets? In 'Applied Survival Analysis,' Hosmer & Lemeshow make reference to Kuk's method, but I cannot find the original paper. Can anyone describe this method, or, even better, a more modern technique? One may assume normally distributed errors.
$\begingroup$
$\endgroup$
4
3 Answers
$\begingroup$
$\endgroup$
4
I've never heard of Kuk's method, but the hot topic these days is L1 minimisation. The rationale being that if you use a penalty term of the absolute value of the regression coefficients, the unimportant ones should go to zero.
These techniques have some funny names: Lasso, LARS, Dantzig selector. You can read the papers, but a good place to start is with Elements of Statistical Learning, Chapter 3.
-
2$\begingroup$ BTW, the penalized R package (j.mp/bdQ0Rp) includes l1/l2 penalized estimation for Generalized Linear and Cox models. $\endgroup$– chlCommented Aug 25, 2010 at 20:07
-
$\begingroup$ stuck in matlab land, implementing it myself... $\endgroup$ Commented Aug 25, 2010 at 23:23
-
$\begingroup$ LARS is great, BTW. very cool stuff. not sure how I can jam it into the framework of Cox Proportional Hazards model, tho... $\endgroup$ Commented Aug 26, 2010 at 16:56
-
3$\begingroup$ The Glmnet software has a lasso'd Cox PH model: cran.r-project.org/web/packages/glmnet/index.html there is also a MATLAB version (not sure if it does a cox model though): www-stat.stanford.edu/~tibs/glmnet-matlab $\endgroup$ Commented Aug 27, 2010 at 9:29
$\begingroup$
$\endgroup$
1
This is a huge topic. As previously mentioned, Hastie, Tibshirani and Friedman give a good intro in Ch3 of Elements of Statistical Learning.
A few points.
What do you mean by "best" or "optimal"? What is best in one sense may not be best in another. Two common criteria are predictive accuracy (predicting the outcome variable) and producing unbiased estimators of the coefficients. Some methods, such as Lasso & Ridge Regression inevitably produce biased coefficient estimators.
The phrase "best subsets" itself can be used in two separate senses. Generally to refer to the best subset among all predictors which optimises some model building criteria. More specifically it can refer to Furnival and Wilson's efficient algorithm for finding that subset among moderate (~50) numbers of linear predictors (Regressions by Leaps and Bounds. Technometrics, Vol. 16, No. 4 (Nov., 1974), pp. 499-51).
answered Aug 26, 2010 at 0:56
-
$\begingroup$ 1) yes, the question is somewhat ambiguous; there are, as you mention, many definitions of 'optimal': via information criterion, cross validation, etc. Most of the heuristic approaches I have seen to the problem proceed by stepwise predictor addition/removal: single pass forward addition or subtraction, etc. However, Hosmer & Lemeshow make reference to this method (a variant of work by Lawless & Singhal), which somehow 'magically' selects predictors by a single computation of an MLR (modulo some other stuff). I am very curious about this method... $\endgroup$ Commented Aug 26, 2010 at 16:55
$\begingroup$
$\endgroup$
2
What I learned it that firstly use Best Subsets Approach as a screening tool, then the stepwise selection procedures can help you finally decide which models might be best subset models (at this time the number of those models is pretty small to handle). If one of the models meets the model conditions, does a good job of summarizing the trend in the data, and most importantly allows you to answer your research question, then congrats your job is done.
-
1$\begingroup$ I think you may be misremembering this. Best subsets is much more computationally expensive than stepwise, but would necessarily catch anything stepwise would, so you would use stepwise to screen & best subsets after. FWIW, I disagree w/ the naive use of these strategies, for reasons I discuss in my answer here: algorithms for automatic model selection. $\endgroup$ Commented Jul 22, 2014 at 17:12
-
$\begingroup$ Stepwise variable selection is a disaster, leading to non-reproducible research. $\endgroup$ Commented Jun 8 at 13:15
penalizedR package), j.mp/cooIT3. Maybe this one too, j.mp/bkDQUj. Cheers $\endgroup$