0
$\begingroup$

I have a very simple model built using Keras.

What strikes me as surprising is that the very same training configs converge (i.e. training loss goes down with every epoch) when the model uses the adagrad optimizer but it diverges when I use adadelta!

Could somebody help me understand what's going on here?

model = Sequential()
model.add(Embedding(input_dim=V, output_dim=dim, input_length=window_size*2))
model.add(Lambda(lambda x: K.mean(x, axis=1), output_shape=(dim,)))
model.add(Dense(V, activation='softmax'))

model.compile(loss='categorical_crossentropy', optimizer='adagrad')

OUTPUT: (epoch, loss)

0 49552.6903033
1 45662.5071292
2 44462.3673022
3 43727.0231725
4 43185.9137167
5 42749.3707656
6 42377.1162052
7 42047.9165432
8 41749.4846331
9 41474.2329881
10 41217.2226644
11 40975.0845323
12 40745.4186415
13 40526.44464
14 40316.7942798
15 40115.3849271
16 39921.3384566
17 39733.9278793
18 39552.543063
19 39376.6627278

Same exact model, using adadelta instead.

model = Sequential()
model.add(Embedding(input_dim=V, output_dim=dim, input_length=window_size*2))
model.add(Lambda(lambda x: K.mean(x, axis=1), output_shape=(dim,)))
model.add(Dense(V, activation='softmax'))

model.compile(loss='categorical_crossentropy', optimizer='adadelta')

OUTPUT: (epoch, loss)

0 48736.8479609
1 44398.2607443
2 44319.0723765
3 44407.3853657
4 44532.3766572
5 44652.6506799
6 44765.2082365
7 44870.688424
8 44968.974878
9 45060.2695123
10 45145.9659934
11 45225.7906622
12 45296.9101175
13 45360.4272417
14 45416.4886399
15 45463.25085
16 45503.1790429
17 45536.4777958
18 45562.5535127
19 45582.7526548
$\endgroup$
1
$\begingroup$

Might be a trivial reason: in Keras Adagrad's optimizer learning rate defaults to 0.01, while Adadelta's default is 1.0. Just try Adadelta with lower learning rate.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.