I have a regression problem where I would like to predict values given several thousand sparse features. The general data set is an $n \times m$ matrix where each row contains a sample with a value I would like to predict. The remaining columns are features that I want to use to predict the value. What would be a good approach to determine which features are good predictors? Thanks for the help.
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
4
A generally safe answer would be to perform a penalized regression. LASSO based penalties would ensure a sparse solution, but an Elastic Net solution would introduce some Ridge penalty and ensure a stable solution. The balance between Lasso and Ridge should likely be determined by your a priori belief of the gathered features importances. If you think they are all important, then go mostly Ridge. If you think only a few are important, then go mostly Lasso. You can try to tune the balance, but it usually doesn't gain you much and can overfit on top of tuning the actual penalty parameter.
The best training algorithm for this method is likely Coordinate Descent / Alternating Least Squares. The mature R implementation is glmnet. The CRAN page for glmnet has a lot of great references for this whole topic. Python also has a similar implementation in it's machine learning toolkit. Both of these platforms let you store your data in an appropriately sparse format and tune the penalty parameter via cross-validation. glmnet allows for case weights and non-gaussian conditional responses. I'm not as knowledgable about the python implementation.
The above describes a mostly linear based solution, so you should screen for outliers in your features. Standard transformations (like square roots of counts) are in order. You can go towards non-linear tree-based solutions, but that didn't sound like what you were after. If it is, then just ask.
(Update many moons later: the now popular xgboost library does implement strong ensemble based tree algorithms on sparse data now: https://github.com/dmlc/xgboost)
answered Sep 17, 2012 at 23:52
-
$\begingroup$ Thanks, this is a big help. Unfortunately, my data not linear. You alluded to a tree-based solution. Could you expand on what algorithms you were thinking of? Thanks again. $\endgroup$– turtleCommented Sep 18, 2012 at 10:51
-
$\begingroup$ I'm not aware of tree based solutions that natively handle sparse matrix input, so I hope your N is 10k or less. With a large number of features, you'll probably want to only try a subset of features at each split. This feature-subspacing is most common in randomForest implementations. R's
randomForestpackage is mature and can provide permutation based feature importance metrics on the backend. If you really want to perform actual feature selection, thecaretorBorutapackages wrap recursive randomForest fits. I wouldn't generally recommend full feature selection however. $\endgroup$ Commented Sep 18, 2012 at 12:52 -
2$\begingroup$ There have been implementations of LASSO for nonlinear data. $\endgroup$ Commented Sep 30, 2012 at 22:20
-
$\begingroup$ As a much later follow up, the now popular
xgboostlibrary does implement strong ensemble based tree models on sparse data now: github.com/dmlc/xgboost $\endgroup$ Commented Feb 3, 2016 at 21:03