1
$\begingroup$

I have generated an adaboost classifier in Weka on a dataset where each instance falls into one of two classes. The result was a number of decision trees, each assigned a weight.

What is the proper method for implementing the classifier generated by adaboost? I assumed the answer was

(Weight of Tree 1 * decision of tree1) + (weight of tree 2 * decision of tree 2) ... + (weight of tree n * decision of tree n)

Where each tree will decide if the instance falls into class A (returning +1) or class B (returning -1)

If the final result sum of weight*result is positive the instance is class A, if negative, class B.

The problem is when I implemented exactly this the results are nowhere near what Weka produced, so I assume that this was not the correct way to implement the classifier.

What should I have done instead?

Improve this question
$\endgroup$
3
  • $\begingroup$ I don't know what Weka does, but usually Adaboost builds a tree, reweighs the data based on the outcomes and uses the reweighed data to build the next tree. This process is repeated. In addition, at every state only a (typically small) multiple of the new tree is added to the answer. Such a process would not produce the formula you describe. $\endgroup$
    – meh
    Commented Apr 15, 2016 at 21:54
  • $\begingroup$ It looks like you have accidentally created a second account and that is stopping you from being able to immediately edit your own question. Have a look at our help centre to see how to merge them. $\endgroup$
    – Silverfish
    Commented Apr 15, 2016 at 22:01
  • $\begingroup$ -aginensky: That's exactly what happened, but I'm not asking a question about how to implement adaboost, but rather how to implement the classifier that adaboost produced. The classifier is presented as a series of trees, each with an assigned weight. Here is an example: youtu.be/ix6IvwbVpw0?t=343 Here we see a number of weighted classifiers. Combine the weights * classifier and check the sum. $\endgroup$
    – djc6535
    Commented Apr 15, 2016 at 22:17

1 Answer 1

Reset to default
1
$\begingroup$

Let $G_m(x) \ m = 1,2,...,M$ be the sequence of weak classifiers, the objective is to build the following:

$$G(x) = \text{sign} \left( \alpha_1 G_1(x) + \alpha_2 G_2(x) + ... \alpha_M G_M(x)\right) = \text{sign} \left( \sum_{m = 1}^M \alpha_m G_m(x)\right)$$

Here the coefficients $\alpha_m$ are computed by the boosting algorithm, and weight the contribution of each respective $G_m(x)$.

The prediction from all classifiers are combined through a weighted majority vote to produce the final prediction.

Source:

  • Elements of statistical learning II - chapter 10
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.