# Decision tree stuck?

I'm trying to create a step-by-step visual of how Adaboost works. I'm training a series of decision tree stumps (tree with depth of 1) and for each iteration, I want to plot that particular stump, showing that the points that were misclassified are now weighted more heavily, and the next classifier should focus on those points. I do this by looping:

for i in range(N):
clf = DecisionTreeClassifier(max_depth=1).fit(X,y, sample_weight=d)
pred = clf.predict(X)
# plot output of pred
# calculate a new d using the Adaboost algorithm


My problem is that after a few iterations, it appears that the Decision tree stump doesn't get re-weighted and continues to predict all observations belonging to one class. Here is what each plot looks like:

Great - got some, missed others. If I continue in this fashion, then after 4 or 5 iterations, I get this:

This particular classifier has classified all the points as one class, and all the rest of the iterations do the same thing.

Can someone explain to me why this is the case with stumps? It doesn't happen when I use trees of greater depth.

EDIT

Here are my X's and y's (a new, smaller dataset than the images):

X's: [[-0.43838916 -1.71793854]
[-1.51794608 -0.34487949]
[-0.77530244  0.74435691]
[ 0.48372805  0.76491468]
[ 0.84696082  1.00027914]
[-1.61250896  0.87429049]
[ 0.92385058 -2.59586644]
[ 0.12134204  1.03762471]]
y's:  [0 0 1 1 1 1 0 0]


Here is the output of my predictions and the re-weighted samples:

Iteration:  0
Weights: [ 0.125  0.125  0.125  0.125  0.125  0.125  0.125  0.125]
Predictions: [0 0 1 1 1 1 0 1]
+++++++++++++++++++++++++++
Iteration:  1
Weights: [ 0.18142703  0.18142703  0.06857297  0.06857297  0.06857297  0.06857297
0.18142703  0.18142703]
Predictions: [0 0 1 1 1 1 0 1]
+++++++++++++++++++++++++++
Iteration:  2
Weights: [ 0.21223494  0.21223494  0.03776506  0.03776506  0.03776506  0.03776506
0.21223494  0.21223494]
Predictions: [0 0 0 0 0 0 0 0]
+++++++++++++++++++++++++++
Iteration:  3
Weights: [ 0.21223494  0.21223494  0.03776506  0.03776506  0.03776506  0.03776506
0.21223494  0.21223494]
Predictions: [0 0 0 0 0 0 0 0]
+++++++++++++++++++++++++++


After only 3 iterations on this dataset it predicts all class 0.

• Do you get negative weights with Ada-boost? – Simone Oct 31 '15 at 2:05
• I don't. They always sum to 1.0 (after I normalize). They just converge to some number and don't change – ilanman Oct 31 '15 at 3:01
• Can you please provide the last datataset you show and the weight vector d? Just print X y and d. You can even print a reduced version as long as you get the same error – Simone Oct 31 '15 at 3:08
• From a quick look your problem starts to appear when your weights get close to 0: eg 0.037 try to multiply your weights by a constant. I will investigate further tomorrow – Simone Oct 31 '15 at 13:34
• Hmmm...I don't know because the weights are normalized, so multiplying them won't change anything (they must sum to 1). – ilanman Oct 31 '15 at 17:50

epsilon = d.dot(pred != new_labels) # calculate error