I'm reading paper about the Adam optimizer and went up until the bias-correction section; in the paper they estimate the bias of the moving average of the squared gradient. These are the equations they use:

enter image description here

Where $v_{t}$ is the moving average of gradient squared, $g_{i}$ is gradient at the $i^{th}$ iteration and $\beta_{2}$ is a moving average parameter.

I don't understand how they make any of these transitions from the first row to the second and on to the third one. Could someone explain what rules are allowing them to do so ?


I don't think the authors do a thorough job of defining everything in their equations explicitly. I'm not sure about the first equality, but as far as I can tell, $\zeta$ is an error term from approximating the gradient's previous values, $g_{i}^{2}$ for $1 \leq i \leq t$, by the most recent value $g_{t}^{2}$. This allows you to move $g_{t}^{2}$ outside of the sum since it no longer depends on $i$, and as all the terms involving $\beta_{i}$ are constant, they can be taken outside of the expectation since they are nonrandom by the linearity of expectation. This is hinted at by the statement following the equations - "where ζ = 0 if the true second moment $E[g_{i}^{2}]$ is stationary." If the second moment is stationary, it is constant at each time step, and so the error is $0$ in making this approximation. That gets you from the first to the second line.

Going from the second to the third line is a more straightforward application of the formula for the sum of a finite geometric series:

$ \sum_{i = 1}^{t} r ^{i} = \frac{r(1-r^{t})}{1 - r}$

Take $r = \beta_{2}^{-1}$ after factoring $\beta_{2}^{t}$ out of the summation, as it does not depend on $i$:

$(1 - \beta_{2})\sum_{i=1}^{t}\beta_{2}^{t-i} =\beta_{2}^{t}(1 - \beta_{2})\sum_{i=1}^{t}(\beta^{-1}_{2})^{i} = \beta_{2}^{t}(1 - \beta_{2})\frac{\beta_{2}^{-1}(1-\beta_{2}^{-t})}{1 - \beta_{2}^{-1}} = (1 - \beta_{2})\frac{(\beta_{2}^{t} - 1)}{ \beta_{2} - 1} = 1 - \beta_{2}^{t} $

Where the third equality follows from multiplying the $\beta_{2}^{t}$ term by the $(1 - \beta_{2}^{-t})$ term and multiplying the numerator and denominator by $\beta_{2}$. The last equality follows from cancelling the $(1 - \beta_{2})$ terms and multiplying the remaining $-1$ by the $(\beta_{2}^{t} - 1)$ term.

  • $\begingroup$ why is possible to approximate g_i with g_t ? by the time we go to the t timestamp, we've already made weight updates, which mean gradients should be different as they are taken from different points in parameter space ? $\endgroup$ – Виталик Бушаев Sep 10 '18 at 9:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.