According to Leon Bottou on page 10 and page 11, he mentions :

Use learning rates of the form $γ_t = γ_0 (1 + γ_0λt)^{−1}$

Im trying to implement it for my neural network but im having trouble understanding the variables...

I think i understood a few but im still confused :

$y_t$ = New Learning Rate

$y_0$ = Initial Learning Rate

$λ$ = L2 regularization ? Is this a pre-fixed value ?

$t$ = ? The current epoch on the training Iteration ?


Bottou, Léon. "Stochastic gradient descent tricks." In Neural networks: Tricks of the trade, pp. 421-436. Springer Berlin Heidelberg, 2012. http://cilvr.cs.nyu.edu/diglib/lsml/bottou-sgd-tricks-2012.pdf ; https://scholar.google.com/scholar?cluster=13393602912095771108&hl=en&as_sdt=0,22


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.