Is there an ensemble classifier that results in sparse solutions for the feature vector like Lasso Regression? With Logistic Regression, I can choose L1 penalization from the penalty hyperparameter. Is my only option to build an ensemble with VotingClassifier to do this or is there an out-of-the-box ensemble classifier in sklearn that is known to produce sparse solutions?

  • 1
    $\begingroup$ I removed a bunch of backticks from your question text. Those are for denoting identifiers or code. $\endgroup$ Oct 11, 2016 at 19:21
  • $\begingroup$ Why do you specifically want an ensemble method? Is lasso regression not doing what you want somehow? $\endgroup$ Oct 11, 2016 at 19:22
  • $\begingroup$ SVM with L1 may be good for your use, but cannot think about a ensemble methods for that. $\endgroup$
    – Haitao Du
    Oct 11, 2016 at 20:02
  • $\begingroup$ Sorry I went trigger happy happy the ticks. Lasso regression is usually for predicting continuous values right? I need something to predict class. Do you mean throwing the output into a logit function? I wanted to use an ensemble method b/c I read that they are much more robust. $\endgroup$
    – O.rka
    Oct 11, 2016 at 20:43

1 Answer 1


Yes, you can use a Logistic Regression model with L1 penalization as your ensemble classifier. We will call that your Level 1 Model. This will give a final predicted class probability for each sample. The features of this ensemble learner (Level 1) are essentially the predicted probabilities for every sample from each of your Level 0 models. In other words, each feature of the Level 1 learner is a model from Level 0. While performing CV training/test maintain the same fold assignments for each level to try and prevent overfitting. In addition, it is imperative while testing an ensemble learner to keep a validation set, or a hold out set, that the Level 0 and Level 1 models have never seen.

Not sure if there is an out of the box solution included with scikit-learn, but easy enough to write your own.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.