gradient boosting can be seen as focusing on the hard examples (the training set examples where the prediction is still far from the true label, and the gradient is still big). is there a similar notion for neural networks? as in selecting the next batch of examples from the hard examples?

(note that this is a different question than Boosting neural networks - where they talk about using neural network as the weak learner)


A few comments on this:

Upweighting hard examples is more a result of gradient boosting rather than the philosophy behind gradient boosting. In gradient boosting using trees as base learners, effectively the second tree tries to find a way to partition the feature space, such that examples which were mis-classified to a similar degree end up in the same node. It then assigns a correction to these guys.

The reason this is necessary, is because when you get to the bottom of a single tree, the mis-classified examples live in different terminal nodes, and are thus separated. You need a new tree to find a different partitioning of the space.

Note that you wouldn't need to do this if you trained trees with no max depth, you could correctly classify all training examples (obviously this would not generalise well).

In general, one finds with tree-based models, that at some point, when you're training a tree, you'll get better results by stopping and training a new one whose goal is to improve the existing tree/ensemble, than going deeper. Both at methods which decrease training loss but the former is generally observed to perform better on test data (i.e. XGBoost performing better than a random forest, especially a random forest with no max depth)

The equivalent situation with a neural network is as follows, you've trained for a bit, your training loss has gone down a fair bit, but you're still mis-classifying some examples in the training set. Conventional wisdom says you could keep gradient descending until your validation loss stops going down, or you could fix the predictions of the network and train a second one on the outputs of the first.

I don't know why one tends to do the former rather than the latter (I'd imagine it's computational reasons), but rest assured that if you did the latter (i.e. boosted a neural network), this would have the same effect as "upweighting" the examples the first network got wrong.

The short answer to the above is that "boosting does not literally mean focussing on hard examples, it means training a classifier/regressor whose target is a measure of distance between the true target variable and the predictions of an existing model. A model that does this will tend to focus on examples the existing model got wrong"

I realise I've rambled a bit here, does this address your question?

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.