3
$\begingroup$

In Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks, they define the weight vector as $$ \mathbf w={g\over\Vert\mathbf v\Vert}\mathbf v $$ Then they differentiate through it to obtain the gradient of a loss function $L$ w.r.t. $\mathbf v$, obtaining $$ \nabla_{\mathbf v}L={g\over \Vert\mathbf v\Vert}\nabla_\mathbf wL-{g\nabla_gL\over \Vert\mathbf v\Vert^2}\mathbf v $$ I don't understand how they obtain the second term, ${g\nabla_gL\over \Vert\mathbf v\Vert^2}\mathbf v$. How can I compute this term?

$\endgroup$

1 Answer 1

3
$\begingroup$

First, we find derivative of $\mathbf{w}$ with respect to $\mathbf{v}$, by sticking with matrix calculus conventions, i.e. $\partial$ notation is different from $\nabla$ notation (i.e. transpose of it, $\nabla_{\mathbf{v}}\mathbf{w}$ is of dimension $d_w\times d_v$, and $\partial\mathbf{w}/\partial \mathbf{v}$ is of dimension $d_v\times d_w$): $$(\nabla_\mathbf{v}\mathbf{w})^T=\frac{\partial\mathbf{w}}{\partial\mathbf{v}}=g\frac{\mathbf{I}}{\Vert\mathbf{v}\Vert}-g\frac{\mathbf{v}\mathbf{v}^T}{\Vert\mathbf{v}\Vert^3}$$ And, execute the chain rule: $$\begin{align}(\nabla_{\mathbf{v}}{L})^T=\frac{\partial L}{\partial \mathbf{v}} &= \frac{\partial L}{\partial \mathbf{w}}\frac{\partial \mathbf{w}}{\partial \mathbf{v}}=\frac{\partial L}{\partial \mathbf{w}}\left(g\frac{\mathbf{I}}{\Vert\mathbf{v}\Vert}-g\frac{\mathbf{v}\mathbf{v}^T}{\Vert\mathbf{v}\Vert^3}\right) \\&=(\nabla_\mathbf{w}L)^T\frac{g}{\Vert\mathbf{v}\Vert}-g(\nabla_\mathbf{w}L)^T\frac{\mathbf{v}\mathbf{v}^T}{\Vert\mathbf{v}\Vert^3}\end{align}$$ Where you can write $(\nabla_\mathbf{w}L)^T\mathbf{v}$ as a dot-product: $\nabla_\mathbf{w}L \cdot \mathbf{v}$, and when substituted we have:

$$\begin{align}(\nabla_{\mathbf{v}}{L})^T=\frac{\partial L}{\partial \mathbf{v}} &=(\nabla_\mathbf{w}L)^T\frac{g}{\Vert\mathbf{v}\Vert}-g\left(\frac{\nabla_\mathbf{w}L\cdot \mathbf{v}}{\Vert \mathbf{v}\Vert}\right)\frac{\mathbf{v}^T}{\Vert\mathbf{v}\Vert^2}\\ &=(\nabla_\mathbf{w}L)^T\frac{g}{\Vert\mathbf{v}\Vert}-g\frac{\nabla_gL}{\Vert\mathbf{v}\Vert^2}\mathbf{v}^T\end{align}$$

And, transposing all yields: $$\nabla_{\mathbf{v}}{L}=\nabla_\mathbf{w}L\frac{g}{\Vert\mathbf{v}\Vert}-g\frac{\nabla_gL}{\Vert\mathbf{v}\Vert^2}\mathbf{v}$$

$\endgroup$
3
  • $\begingroup$ Thanks for answering, but I do not understand how to differentiate through $1\over \Vert\mathbf v\Vert$ and get $g{\mathbf v\mathbf v^T\over \Vert\mathbf v\Vert^3}$? $\endgroup$
    – Maybe
    May 5, 2019 at 23:31
  • $\begingroup$ @SherwinChen you can find it in most of the matrix calculus cheatsheets. $\endgroup$
    – gunes
    May 6, 2019 at 0:39
  • 1
    $\begingroup$ Thank you, I found it here, Eq.130. Derivation can be found here $\endgroup$
    – Maybe
    May 6, 2019 at 1:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.