what is instance weight in boosting?

Question

Hi, I am reading something about Boosting, and I had hard time understanding one of the steps in boosting - assign greater weights to those instances.

What does the sentence - assign greater weights to those instances mean ? My understanding is.. for example

Initially, we have training data ($x_1$,$y_1$) ($x_2$,$y_2$) ($x3$,$y_3$) ($x_4$,$y_4$) ($x_5$,$y_5$) after we first apply the weak learner, we find that (x2,y2) (x3,y3) are misclassified, and we try to adjust the training data, "assigning the weights" so that new training data become... like ($x_1$,$y_1$) ($x_2$,$y_2$),($x_2$,$y_2$) ($x_3$,$y_3$) ($x_3$,$y_3$) ($x_4$,$y_4$) ($x_5$,$y_5$), where we have more misclassified instances ? thus, next learner have more chances to learn the misclassified ones ?

Answer 1

"Instance" is just a somewhat confusing way of saying "case" or "person" or "observation," etc.

Imagine we have N data points we are trying to predict; each of those data points would be an "instance." If our data look like:

  y x
1 1 4
2 0 2
3 0 3
4 1 3
5 1 3

Then we have 5 "instances" and each row (observation, case, etc.) represents an instance. Imagine we predict y from x using a weak learner. We find that instance #3 (y = 0, x = 3) is classified incorrectly. In the next iteration, we would weight that instance higher than the others.

I wouldn't necessarily say that the learner "has more chances to learn the misclassified ones," as every instance/case/row/observation is included in each iteration. It is just that subsequent learners focus more on misclassified instances.

Answer 2

One important information is probably kept from the text: “The misclassified instances are assigned with a greater weight when computing information gain.”

It simply means that these instances have a greater contribution when computing model loss. So, the gradients you compute to modifying the model weights are more influenced by these instances so the resulting model can correctly classify them.

In other words, the loss computation is no longer an average of individual losses but a weighted average of individual losses - the model receives greater punishment for misclassifying some instances than others.