In Adaboost, each sample is given a weight and the machine learning model will be trained with these weights. I want to use logistic regression model in Adaboost, but how can i use these weights in logistic regression? I mean the classic logistic regression model do not use the weight of sample, how can I change the model to make the weight useful?

Any help would be appreciated!


Fitting to a weighted sample works the same way for pretty much any statistical/machine learning model: you minimize the sample weighted loss function. In the case of logistic regression, you would minimize the sample weighted log-loss:

$$ \sum_i w_i y_i \log(p_i) + w_i (1 - y_i) \log(1 - p_i) $$

Here $w_i$ are the sample weights.

That said, there are two points I should make:

1) Logistic regression is not a hard classifier, while classical AdaBoost assumes your weak learners are, so you will have to pick some threshold on the predicted probabilities of your constituent logistic models.

2) You may be better off just using gradient boosting to minimize the log-loss (i.e. gradient boosted logistic regression). If you are not implementing the model for educational purposes, this is definately something you should consider.

  • $\begingroup$ Thank you for your answer and advice. If the weight is used as your says, when i renew weights in Adaboost, for example weight0 for the first sample, if the sample is classified wrongly, so I need to drag down weight0, am i right? i think this is not the same as Adboost usually do, i.e raise the weight of the sample which is classified wrongly. $\endgroup$ – David Lee Aug 25 '17 at 6:45
  • $\begingroup$ I don't think so. I believe you can use the same basic outline as tree based adaboost, just replace each occurrence of the word "tree" with "logistic regression". In particular, wrongly classified data should have weights increased. $\endgroup$ – Matthew Drury Aug 25 '17 at 14:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.