I'm using the LASSO method, in the problem of text classification (sentiment classification). The features I'm using are mainly Ngrams (every N consecutive words) and I'm using the LASSO specifically so that I can rank the features and extract the set of the significant Ngrams in the classification problem.

My question is about tuning the alpha parameter in the scikitlearn model: I understand that as I set alpha closer to 1, the number of features selected becomes fewer. So I wanted to ask:

  1. What is the best practice to select the number of the important features, hence alpha value (cross validation could be possible if I seek maximum score not model interpretation), but is there's something to measure the "minimum adequate number of features for the classification process"?
  2. If I decided that I would like to select only the top 1000 features and set alpha to return 1000 features with non zero coefficients. Would the LASSO method here differ from using normal linear regression and rank the top 1000 features?
  • $\begingroup$ Minor point: you can actually set alpha to any positive number, including ones above 1. Your model becomes more sparse as alpha increases rather than as it approaches 1. $\endgroup$
    – Ben Kuhn
    Commented Dec 6, 2014 at 17:29

2 Answers 2


First: trying to set alpha to find a pre-specified number of important features isn't a good idea. Whether a feature is predictive of the response is a property of the data, not your model. So you want your model to tell you how many features are important, not the other way around. If you try to mess with your alpha until it finds a pre-specified number of features to be predictive, you run the risk of overfitting (if there are really fewer predictive features than that) or underfitting (if there are more).

This is why the tuning parameter is often selected automatically based on minimizing cross-validated generalization error. In the cross-validation setting, people frequently do something similar to finding the "minimum adequate number of features", which is to select the largest alpha where the error is at most one standard deviation above the alpha parameter with the lowest cross-validated error (e.g. here, p. 18). The rationale for this is that there's some noise in the cross-validated error estimate, and if you select the alpha that simply minimizes the estimate you risk overfitting to the noise, so it's better to "err on the side of parsimony," as the paper puts it.

On the other hand, some papers (e.g. here) have noted that selecting alpha by minimizing cross-validated error does not yield consistent feature selection in practice (i.e., where features are selected if and only if they should be). An alternative is selecting based on the BIC, as advocated by e.g. Zou, Hastie and Tibshirani here. (For the BIC one should set the "degrees of freedom" equal to the rank of the feature matrix for the features found to be nonzero; see the paper for more detail.)

  • $\begingroup$ since we will inspect the feature and manually filter afterwards probably i'll go for the cross-validation method. Thanks also for the comments about overfitting, it's useful always to remind oneself about that. $\endgroup$ Commented Dec 6, 2014 at 21:21

You could try applying the Elastic Net Regression some times. It combines the Lasso and Ridge regression methods in order to give your feature selection a 'human touch'. This is quite helpful in optimizing your features while preserving the intuition from features in your data.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.