Well recently I was working on learning boosting algorithms, such as adaboost, gradient boost, and I have known the fact that the most common used weak-learner is trees. I really want to know are there some recent successful examples (I mean some papers or articles) for using neural networks as base learner.


2 Answers 2


In boosting, weak or or unstable classifiers are used as base learners. This is the case because the aim is to generate decision boundaries that are considerably different. Then, a good base learner is one that is highly biased, in other words, the output remains basically the same even when the training parameters for the base learners are changed slightly.

In neural networks, dropout is a regularization technique that can be compared to training ensembles. The difference is that the ensembling is done in the latent space (neurons exist or not) thus decreasing the generalization error.

"Each training example can thus be viewed as providing gradients for a different, randomly sampled architecture, so that the final neural network efficiently represents a huge ensemble of neural networks, with good generalization capability" - quoting from here.

There are two such techniques: in dropout neurons are dropped (meaning the neurons exist or not with a certain probability) while in dropconnect the weights are dropped.

Now, to answer your question, I believe that neural networks (or perceptrons) are not used as base learners in a boosting setup since they are slower to train (just takes too much time) and the learners are not as weak, although they could be setup to be more unstable. So, it's not worth the effort.

There might have been research on this topic, however it's a pity that ideas that don't work well are not usually successfully published. We need more research covering pathways that don't lead anywhere, aka "don't bother trying this".


I had a bit more though on this and if you are interested in ensembles of large networks, then you might be referring to methods of combining the outputs of multiple such networks. Most people average or use majority voting depending on the task - this might not be optimal. I believe it should be possible to change the weights for each network's output according to the error on a particular record. The less correlated the outputs, the better your ensembling rule.

  • $\begingroup$ hi, why base learners must be weak and biased? $\endgroup$
    – dingx
    Commented May 23, 2023 at 6:48

I see this has does not have an accepted answer so I'll give a very heuristic answer. Yes, it is done....e.g. it is available in JMP Pro (probably the best stat package you've never heard of). http://www.jmp.com/support/help/Overview_of_Neural_Networks.shtml

There's a description in in the middle of the page of what it is used for. I haven't put any cycles into investigating the theory, but it seems they are implying it achieves essentially the same results as using more nodes in a single larger model. The advantage [they claim] is in speed of model fitting.

For just a very rough gauge, I compared it on a dataset I have with 2 sigmoid and 2 Gaussian nodes and boosting the model 6x against 12 sigmoid and 12 Gaussian nodes in a single model and the results were virtually identical on my test set of data.

I didn't notice any speed difference either...but the dataset is only 1600 points and I'm only using 12 variables, so on a larger dataset with more variables it may hold true that there is a noticeable computation difference.


Not the answer you're looking for? Browse other questions tagged or ask your own question.