In Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks, they define the weight vector as $$\mathbf w={g\over\Vert\mathbf v\Vert}\mathbf v$$ Then they differentiate through it to obtain the gradient of a loss function $$L$$ w.r.t. $$\mathbf v$$, obtaining $$\nabla_{\mathbf v}L={g\over \Vert\mathbf v\Vert}\nabla_\mathbf wL-{g\nabla_gL\over \Vert\mathbf v\Vert^2}\mathbf v$$ I don't understand how they obtain the second term, $${g\nabla_gL\over \Vert\mathbf v\Vert^2}\mathbf v$$. How can I compute this term?

First, we find derivative of $$\mathbf{w}$$ with respect to $$\mathbf{v}$$, by sticking with matrix calculus conventions, i.e. $$\partial$$ notation is different from $$\nabla$$ notation (i.e. transpose of it, $$\nabla_{\mathbf{v}}\mathbf{w}$$ is of dimension $$d_w\times d_v$$, and $$\partial\mathbf{w}/\partial \mathbf{v}$$ is of dimension $$d_v\times d_w$$): $$(\nabla_\mathbf{v}\mathbf{w})^T=\frac{\partial\mathbf{w}}{\partial\mathbf{v}}=g\frac{\mathbf{I}}{\Vert\mathbf{v}\Vert}-g\frac{\mathbf{v}\mathbf{v}^T}{\Vert\mathbf{v}\Vert^3}$$ And, execute the chain rule: \begin{align}(\nabla_{\mathbf{v}}{L})^T=\frac{\partial L}{\partial \mathbf{v}} &= \frac{\partial L}{\partial \mathbf{w}}\frac{\partial \mathbf{w}}{\partial \mathbf{v}}=\frac{\partial L}{\partial \mathbf{w}}\left(g\frac{\mathbf{I}}{\Vert\mathbf{v}\Vert}-g\frac{\mathbf{v}\mathbf{v}^T}{\Vert\mathbf{v}\Vert^3}\right) \\&=(\nabla_\mathbf{w}L)^T\frac{g}{\Vert\mathbf{v}\Vert}-g(\nabla_\mathbf{w}L)^T\frac{\mathbf{v}\mathbf{v}^T}{\Vert\mathbf{v}\Vert^3}\end{align} Where you can write $$(\nabla_\mathbf{w}L)^T\mathbf{v}$$ as a dot-product: $$\nabla_\mathbf{w}L \cdot \mathbf{v}$$, and when substituted we have:
\begin{align}(\nabla_{\mathbf{v}}{L})^T=\frac{\partial L}{\partial \mathbf{v}} &=(\nabla_\mathbf{w}L)^T\frac{g}{\Vert\mathbf{v}\Vert}-g\left(\frac{\nabla_\mathbf{w}L\cdot \mathbf{v}}{\Vert \mathbf{v}\Vert}\right)\frac{\mathbf{v}^T}{\Vert\mathbf{v}\Vert^2}\\ &=(\nabla_\mathbf{w}L)^T\frac{g}{\Vert\mathbf{v}\Vert}-g\frac{\nabla_gL}{\Vert\mathbf{v}\Vert^2}\mathbf{v}^T\end{align}
And, transposing all yields: $$\nabla_{\mathbf{v}}{L}=\nabla_\mathbf{w}L\frac{g}{\Vert\mathbf{v}\Vert}-g\frac{\nabla_gL}{\Vert\mathbf{v}\Vert^2}\mathbf{v}$$
• Thanks for answering, but I do not understand how to differentiate through $1\over \Vert\mathbf v\Vert$ and get $g{\mathbf v\mathbf v^T\over \Vert\mathbf v\Vert^3}$? – Maybe May 5 '19 at 23:31