In Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks, they define the weight vector as $$ \mathbf w={g\over\Vert\mathbf v\Vert}\mathbf v $$ Then they differentiate through it to obtain the gradient of a loss function $L$ w.r.t. $\mathbf v$, obtaining $$ \nabla_{\mathbf v}L={g\over \Vert\mathbf v\Vert}\nabla_\mathbf wL-{g\nabla_gL\over \Vert\mathbf v\Vert^2}\mathbf v $$ I don't understand how they obtain the second term, ${g\nabla_gL\over \Vert\mathbf v\Vert^2}\mathbf v$. How can I compute this term?


First, we find derivative of $\mathbf{w}$ with respect to $\mathbf{v}$, by sticking with matrix calculus conventions, i.e. $\partial$ notation is different from $\nabla$ notation (i.e. transpose of it, $\nabla_{\mathbf{v}}\mathbf{w}$ is of dimension $d_w\times d_v$, and $\partial\mathbf{w}/\partial \mathbf{v}$ is of dimension $d_v\times d_w$): $$(\nabla_\mathbf{v}\mathbf{w})^T=\frac{\partial\mathbf{w}}{\partial\mathbf{v}}=g\frac{\mathbf{I}}{\Vert\mathbf{v}\Vert}-g\frac{\mathbf{v}\mathbf{v}^T}{\Vert\mathbf{v}\Vert^3}$$ And, execute the chain rule: $$\begin{align}(\nabla_{\mathbf{v}}{L})^T=\frac{\partial L}{\partial \mathbf{v}} &= \frac{\partial L}{\partial \mathbf{w}}\frac{\partial \mathbf{w}}{\partial \mathbf{v}}=\frac{\partial L}{\partial \mathbf{w}}\left(g\frac{\mathbf{I}}{\Vert\mathbf{v}\Vert}-g\frac{\mathbf{v}\mathbf{v}^T}{\Vert\mathbf{v}\Vert^3}\right) \\&=(\nabla_\mathbf{w}L)^T\frac{g}{\Vert\mathbf{v}\Vert}-g(\nabla_\mathbf{w}L)^T\frac{\mathbf{v}\mathbf{v}^T}{\Vert\mathbf{v}\Vert^3}\end{align}$$ Where you can write $(\nabla_\mathbf{w}L)^T\mathbf{v}$ as a dot-product: $\nabla_\mathbf{w}L \cdot \mathbf{v}$, and when substituted we have:

$$\begin{align}(\nabla_{\mathbf{v}}{L})^T=\frac{\partial L}{\partial \mathbf{v}} &=(\nabla_\mathbf{w}L)^T\frac{g}{\Vert\mathbf{v}\Vert}-g\left(\frac{\nabla_\mathbf{w}L\cdot \mathbf{v}}{\Vert \mathbf{v}\Vert}\right)\frac{\mathbf{v}^T}{\Vert\mathbf{v}\Vert^2}\\ &=(\nabla_\mathbf{w}L)^T\frac{g}{\Vert\mathbf{v}\Vert}-g\frac{\nabla_gL}{\Vert\mathbf{v}\Vert^2}\mathbf{v}^T\end{align}$$

And, transposing all yields: $$\nabla_{\mathbf{v}}{L}=\nabla_\mathbf{w}L\frac{g}{\Vert\mathbf{v}\Vert}-g\frac{\nabla_gL}{\Vert\mathbf{v}\Vert^2}\mathbf{v}$$

| cite | improve this answer | |
  • $\begingroup$ Thanks for answering, but I do not understand how to differentiate through $1\over \Vert\mathbf v\Vert$ and get $g{\mathbf v\mathbf v^T\over \Vert\mathbf v\Vert^3}$? $\endgroup$ – Maybe May 5 '19 at 23:31
  • $\begingroup$ @SherwinChen you can find it in most of the matrix calculus cheatsheets. $\endgroup$ – gunes May 6 '19 at 0:39
  • 1
    $\begingroup$ Thank you, I found it here, Eq.130. Derivation can be found here $\endgroup$ – Maybe May 6 '19 at 1:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.