If you look at the documentation http://keras.io/optimizers/ there is a parameter in the SGD for decay. I know this reduces the learning rate over time. However, I can not figure out how it works exactly. Is it a value which is multiplied by the learning rate such as lr = lr * (1 - decay) is it exponential? Also how can I see what learning rate my model is using? When I print model.optimizer.lr.get_value() after running a fit over a few epochs it gives back the original learning rate even though I set the decay.

Also do I have to set nesterov=True to use momentum or are there just two different types of momentum I can use. For instance is there a point to doing this sgd = SGD(lr = 0.1, decay = 1e-6, momentum = 0.9, nesterov = False)

  • 1
    this question is probably better suited for stackoverflow – lejlot May 7 '16 at 16:46
  • Ok. I will move it when I get home. – chasep255 May 7 '16 at 16:46
up vote 16 down vote accepted

The documentation that you're referring to includes a reference to the Python source (just click on the [Source] link in the appropriate place), that can be used to answer your questions. Here's the most relevant line, showing how decay modifies the learning rate:

lr = self.lr * (1. / (1. + self.decay * self.iterations))

The nesterov option does not have to be set to True for momentum to be used; it results in momentum being used in a different way, as again can be seen from the source:

v = self.momentum * m - lr * g  # velocity

if self.nesterov:
    new_p = p + self.momentum * v - lr * g
else:
    new_p = p + v

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.