Advice for a sparse high-dimensional regression strategy I have a regression problem where I would like to predict values given several thousand sparse features. The general data set is an $n \times m$ matrix where each row contains a sample with a value I would like to predict. The remaining columns are features that I want to use to predict the value. What would be a good approach to determine which features are good predictors? Thanks for the help. 
 A: A generally safe answer would be to perform a penalized regression.  LASSO based penalties would ensure a sparse solution, but an Elastic Net solution would introduce some Ridge penalty and ensure a stable solution.  The balance between Lasso and Ridge should likely be determined by your a priori belief of the gathered features importances.  If you think they are all important, then go mostly Ridge.  If you think only a few are important, then go mostly Lasso.  You can try to tune the balance, but it usually doesn't gain you much and can overfit on top of tuning the actual penalty parameter.
The best training algorithm for this method is likely Coordinate Descent / Alternating Least Squares.  The mature R implementation is glmnet.  The CRAN page for glmnet has a lot of great references for this whole topic.  Python also has a similar implementation in it's machine learning toolkit.  Both of these platforms let you store your data in an appropriately sparse format and tune the penalty parameter via cross-validation.  glmnet allows for case weights and non-gaussian conditional responses.  I'm not as knowledgable about the python implementation.
The above describes a mostly linear based solution, so you should screen for outliers in your features.  Standard transformations (like square roots of counts) are in order.  You can go towards non-linear tree-based solutions, but that didn't sound like what you were after.  If it is, then just ask.
(Update many moons later: the now popular xgboost library does implement strong ensemble based tree algorithms on sparse data now: https://github.com/dmlc/xgboost)
