I have a question about Boosting and stacking in machine learning. Suppose that I will train neural network, SVM and logistic regression using optimization algorithm to optimize best inputs in first phase. So now I have 3 trained models with different combination of inputs. Can I use these trained models (based on different input's variables of each created model) for boosting or stacking?


I am not sure if you are familiar with this topic. If not, read Ensemble learning on Wikipedia.

Now, to your question. First boosting. Here, you use a meta-algorithm to improve an (one) algorithm. It usually does a good job, but sometimes it over-fits, meaning that It appears to be better, but it actually isn't. For your problem - yes, you can use boosting on each of the used algorithms.

As for stacking - here, you run all of the algorithms at once, and then make a decision which of them you will use for the final decision (somewhat similar to voting). You usually make this decision with logistic regression (in your case, you will have two different instances of logistic regression as you already use one for solving your problem).

If you want to use both boosting and stacking, you first have to boost each and every one of your algorithms, and then use another algorithm (e.g. logistic regression) to stack them into an ensemble.

| cite | improve this answer | |
  • $\begingroup$ Thank you for your answer alesc. I don't want use both boosting and stacking together. My general question : Can we combine models (For example using boosting or stacking or...-Ensemble learning) that have different input variables? or it is forbidden based on theory of ensemble learning? I think based on your answer, the answer is yes. we can use. Is this true? $\endgroup$ – user2991243 Feb 28 '15 at 10:07
  • 2
    $\begingroup$ Oh, I forgot to comment on that part. You usually use the same input for all of the algorithms in an ensemble, but I don't see a reason why It wouldn't work. But I can't say anything regarding if it would be better or not. Do you have a good reason not to use the same input? Did you perhaps perform a Feature selection and come with different attributes for different algorithms? $\endgroup$ – alesc Feb 28 '15 at 10:11
  • $\begingroup$ That's true. I used a feature selection in first phase because there isn't any good theory for selecting inputs in my problem. So I have different inputs in every created model. For example I have separate trained models NN,SVM & LR with differed selected inputs. I want know how can I use ensemble learning in this situation to combine all models. $\endgroup$ – user2991243 Feb 28 '15 at 10:13
  • 1
    $\begingroup$ In that case, it should work fine. Because removing some attributes was a kind of mini-boosting for each of your algorithm. Are there major differences between the algorithms or do they perform similar? If one is significantly better than the others, then maybe you don't even need to make an ensemble. $\endgroup$ – alesc Feb 28 '15 at 10:16
  • $\begingroup$ The accuracies are almost the same. For instance 87% for NN,82% for SVM and 80% for LR but type-1 error is so impotent for me. I'm coding a GUI and I want use ensemble learning as a feature in this software + I used 10-fold cross validation for creating every model and an optimization algorithm to optimize and select input variables (feature selection). $\endgroup$ – user2991243 Feb 28 '15 at 10:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.