Neural Network General Learning Dynamics of Gradient Descent This might be simple to you but can someone tell me step by step how is matrix form of updating rule of $W^{32}$ and $W^{21}$ derived in this case?
Consider linear three layer neural network model which has input $\mathbf{x}$, first layer to second layer weight matrix $W^{21}$, second layer to third layer weight matrix $W^{32}$. The output $\mathbf{y}$ is defined as:
$$\mathbf{y} = W^{32}W^{21}\mathbf{x}$$
consider P examples $\{\mathbf{x}^\mu,\mathbf{y}^\mu\}, \mu=1\cdots P$.
Training is accomplished via gradient descent on the squared error $\sum_{\mu=1}^P||\mathbf{y}^\mu-W^{32}W^{21}\mathbf{x^\mu}||^2$ between the desired feature output, and the network’s feature output. This gradient descent procedure yields the batch learning rule.
$$\Delta W^{21} = \lambda\sum_{\mu=1}^PW^{32^T}(\mathbf{y}^\mu\mathbf{x}^{\mu^T} - W^{32}W^{21}\mathbf{x}^\mu\mathbf{x}^{\mu^T})$$
$$\Delta W^{32} = \lambda\sum_{\mu=1}^P(\mathbf{y}^\mu\mathbf{x}^{\mu^T} - W^{32}W^{21}\mathbf{x}^\mu\mathbf{x}^{\mu^T})W^{21^T}$$
where $\lambda$ is learning rate.
 A: General squared error $||y-Wx||^2=(y-Wx)^\top(y-Wx)$. (you missed a square in you sq error formula.)
Expand out your 3 layer error formula using inner products, then refer to matrix calculus formula, for example $\frac{\mathrm{d}y^\top Wx}{\mathrm{d}W} = xy^\top$
A: According to what jf328 hinted. 
Here we expand the function:
$\sum_{\mu=1}^P||\mathbf{y}^\mu-W^{32}W^{21}\mathbf{x^\mu}||^2$
to be 
$$
\begin{equation}
\sum_{\mu=1}^P(\mathbf{y}^\mu-W^{32}W^{21}\mathbf{x^\mu})^T(\mathbf{y}^\mu-W^{32}W^{21}\mathbf{x^\mu})
\end{equation}
$$
Now consider equation:
$$
\begin{equation}
(\mathbf{y}-W^{32}W^{21}\mathbf{x})^T(\mathbf{y}-W^{32}W^{21}\mathbf{x})
\end{equation}
$$ 
expanding function results to:
$$
\begin{equation}
\mathbf{y}^T\mathbf{y} - 2 \mathbf{y}^TW^{32}W^{21}\mathbf{x} + \mathbf{x}^TW^{21} W^{32^T} W^{32}W^{21}\mathbf{x}
\end{equation}
$$
according to matrix differential equation fomula:
$$
\frac{\partial y^\top Wx}{\partial W} = xy^\top 
$$
and 
$$
\frac{\partial b^TX^TDXc}{\partial X} = D^TXbc^T + DXcb^T
$$
$$
\begin{equation}
\frac{\partial (\mathbf{y}^T\mathbf{y} - 2 \mathbf{y}^TW^{32}W^{21}\mathbf{x} + \mathbf{x}^TW^{21^T} W^{32^T} W^{32}W^{21}\mathbf{x})}{\partial W^{21}}
\end{equation}
$$ 
This results to:
$$
0 + 2\mathbf{x}\mathbf{y}^TW^{32} + 2 W^{32^T}W^{32}W^{21}\mathbf{x}\mathbf{x}^T
$$
as
$$
\mathbf{x}\mathbf{y}^T W^{32} = W^{32^T}\mathbf{y}\mathbf{x}^T
$$
previous results become
$$
2W^{32^T}(\mathbf{y}^\mu\mathbf{x}^{\mu^T} - W^{32}W^{21}\mathbf{x}^\mu\mathbf{x}^{\mu^T})$$
Then we finally get:
$$\Delta W^{21} = \lambda\sum_{\mu=1}^PW^{32^T}(\mathbf{y}^\mu\mathbf{x}^{\mu^T} - W^{32}W^{21}\mathbf{x}^\mu\mathbf{x}^{\mu^T})$$
the same for $\Delta W^{32}$
