I have the following vowpal wabbit log. To me it looks quite counter-intuitive: the objective function (l1-regularized hinge loss) seems to go down then suddenly spiking up. I am aware that gradient descent can diverge if the learning rate is too high. What bugs me is this behavior: it tends to converge, then finally diverges. Is this possible for convex functions (it is convex, right?) If the learning rate is not too high initially, and it's decreasing as well as the gradient must, how can it suddenly become too high? What am I missing? Can the vw learning rate decay have something to do with it?
And btw, what's probably wrong with my settings/trainset according to the log? Thanks for your attention.
using l1 regularization = 7.498e-05
final_regressor = /home/mm/models/52.model
Num weight bits = 28
learning rate = 2
initial_t = 0
power_t = 0.5
decay_learning_rate = 1
creating cache_file = /tmp/trtmp.cache
Reading datafile = /tmp/trtmp
num sources = 1
average since example example current current current
loss last counter weight label predict features
0.504495 0.504495 23 2.0 1.0000 -1.0000 16
0.250552 0.000000 68 4.0 -1.0000 -1.0000 15
0.241113 0.232359 143 8.3 1.0000 -1.0000 14
0.311833 0.380056 250 16.9 1.0000 -1.0000 14
0.340020 0.368193 436 33.8 -1.0000 -1.0000 3
0.346579 0.353136 915 67.6 -1.0000 -1.0000 25
0.301540 0.256500 1830 135.2 -1.0000 -1.0000 11
0.253466 0.205517 3625 270.7 1.0000 -1.0000 17
0.225351 0.197240 7251 541.5 -1.0000 -1.0000 2
0.193044 0.160737 14823 1083.1 -1.0000 -1.0000 4
0.171241 0.149439 30096 2166.2 -1.0000 -1.0000 13
0.156050 0.140859 60235 4332.4 -1.0000 -1.0000 3
0.129873 0.103696 120171 8664.8 -1.0000 -1.0000 14
0.110798 0.091725 240159 17330.0 1.0000 1.0000 16
0.104312 0.097826 479570 34660.0 -1.0000 -1.0000 2
0.116619 0.128926 959076 69320.0 -1.0000 -1.0000 18
0.142526 0.168433 1918859 138640.0 -1.0000 -1.0000 14
finished run
number of examples per pass = 62866
passes used = 50
weighted example sum = 227125
weighted label sum = -45425.1
average loss = 0.164373
best constant = -0.2
total feature number = 39394300