Question about the gradient of weight normalization In Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks, they define the weight vector as
$$
\mathbf w={g\over\Vert\mathbf v\Vert}\mathbf v
$$
Then they differentiate through it to obtain the gradient of a loss function $L$ w.r.t. $\mathbf v$, obtaining
$$
\nabla_{\mathbf v}L={g\over \Vert\mathbf v\Vert}\nabla_\mathbf wL-{g\nabla_gL\over \Vert\mathbf v\Vert^2}\mathbf v
$$
I don't understand how they obtain the second term, ${g\nabla_gL\over \Vert\mathbf v\Vert^2}\mathbf v$. How can I compute this term?
 A: First, we find derivative of $\mathbf{w}$ with respect to $\mathbf{v}$, by sticking with matrix calculus conventions, i.e. $\partial$ notation is different from $\nabla$ notation (i.e. transpose of it, $\nabla_{\mathbf{v}}\mathbf{w}$ is of dimension $d_w\times d_v$, and $\partial\mathbf{w}/\partial \mathbf{v}$ is of dimension $d_v\times d_w$):
$$(\nabla_\mathbf{v}\mathbf{w})^T=\frac{\partial\mathbf{w}}{\partial\mathbf{v}}=g\frac{\mathbf{I}}{\Vert\mathbf{v}\Vert}-g\frac{\mathbf{v}\mathbf{v}^T}{\Vert\mathbf{v}\Vert^3}$$
And, execute the chain rule:
$$\begin{align}(\nabla_{\mathbf{v}}{L})^T=\frac{\partial L}{\partial \mathbf{v}} &= \frac{\partial L}{\partial \mathbf{w}}\frac{\partial \mathbf{w}}{\partial \mathbf{v}}=\frac{\partial L}{\partial \mathbf{w}}\left(g\frac{\mathbf{I}}{\Vert\mathbf{v}\Vert}-g\frac{\mathbf{v}\mathbf{v}^T}{\Vert\mathbf{v}\Vert^3}\right) \\&=(\nabla_\mathbf{w}L)^T\frac{g}{\Vert\mathbf{v}\Vert}-g(\nabla_\mathbf{w}L)^T\frac{\mathbf{v}\mathbf{v}^T}{\Vert\mathbf{v}\Vert^3}\end{align}$$
Where you can write $(\nabla_\mathbf{w}L)^T\mathbf{v}$ as a dot-product: $\nabla_\mathbf{w}L \cdot \mathbf{v}$, and when substituted we have:
$$\begin{align}(\nabla_{\mathbf{v}}{L})^T=\frac{\partial L}{\partial \mathbf{v}} &=(\nabla_\mathbf{w}L)^T\frac{g}{\Vert\mathbf{v}\Vert}-g\left(\frac{\nabla_\mathbf{w}L\cdot \mathbf{v}}{\Vert \mathbf{v}\Vert}\right)\frac{\mathbf{v}^T}{\Vert\mathbf{v}\Vert^2}\\ &=(\nabla_\mathbf{w}L)^T\frac{g}{\Vert\mathbf{v}\Vert}-g\frac{\nabla_gL}{\Vert\mathbf{v}\Vert^2}\mathbf{v}^T\end{align}$$
And, transposing all yields:
$$\nabla_{\mathbf{v}}{L}=\nabla_\mathbf{w}L\frac{g}{\Vert\mathbf{v}\Vert}-g\frac{\nabla_gL}{\Vert\mathbf{v}\Vert^2}\mathbf{v}$$
