This might be simple to you but can someone tell me step by step how is matrix form of updating rule of $W^{32}$ and $W^{21}$ derived in this case?

Consider linear three layer neural network model which has input $\mathbf{x}$, first layer to second layer weight matrix $W^{21}$, second layer to third layer weight matrix $W^{32}$. The output $\mathbf{y}$ is defined as: $$\mathbf{y} = W^{32}W^{21}\mathbf{x}$$

consider P examples $\{\mathbf{x}^\mu,\mathbf{y}^\mu\}, \mu=1\cdots P$.

Training is accomplished via gradient descent on the squared error $\sum_{\mu=1}^P||\mathbf{y}^\mu-W^{32}W^{21}\mathbf{x^\mu}||^2$ between the desired feature output, and the network’s feature output. This gradient descent procedure yields the batch learning rule.

$$\Delta W^{21} = \lambda\sum_{\mu=1}^PW^{32^T}(\mathbf{y}^\mu\mathbf{x}^{\mu^T} - W^{32}W^{21}\mathbf{x}^\mu\mathbf{x}^{\mu^T})$$

$$\Delta W^{32} = \lambda\sum_{\mu=1}^P(\mathbf{y}^\mu\mathbf{x}^{\mu^T} - W^{32}W^{21}\mathbf{x}^\mu\mathbf{x}^{\mu^T})W^{21^T}$$

where $\lambda$ is learning rate.


2 Answers 2


General squared error $||y-Wx||^2=(y-Wx)^\top(y-Wx)$. (you missed a square in you sq error formula.)

Expand out your 3 layer error formula using inner products, then refer to matrix calculus formula, for example $\frac{\mathrm{d}y^\top Wx}{\mathrm{d}W} = xy^\top$


According to what jf328 hinted.

Here we expand the function:


to be

$$ \begin{equation} \sum_{\mu=1}^P(\mathbf{y}^\mu-W^{32}W^{21}\mathbf{x^\mu})^T(\mathbf{y}^\mu-W^{32}W^{21}\mathbf{x^\mu}) \end{equation} $$ Now consider equation:

$$ \begin{equation} (\mathbf{y}-W^{32}W^{21}\mathbf{x})^T(\mathbf{y}-W^{32}W^{21}\mathbf{x}) \end{equation} $$

expanding function results to:

$$ \begin{equation} \mathbf{y}^T\mathbf{y} - 2 \mathbf{y}^TW^{32}W^{21}\mathbf{x} + \mathbf{x}^TW^{21} W^{32^T} W^{32}W^{21}\mathbf{x} \end{equation} $$

according to matrix differential equation fomula:

$$ \frac{\partial y^\top Wx}{\partial W} = xy^\top $$


$$ \frac{\partial b^TX^TDXc}{\partial X} = D^TXbc^T + DXcb^T $$

$$ \begin{equation} \frac{\partial (\mathbf{y}^T\mathbf{y} - 2 \mathbf{y}^TW^{32}W^{21}\mathbf{x} + \mathbf{x}^TW^{21^T} W^{32^T} W^{32}W^{21}\mathbf{x})}{\partial W^{21}} \end{equation} $$

This results to:

$$ 0 + 2\mathbf{x}\mathbf{y}^TW^{32} + 2 W^{32^T}W^{32}W^{21}\mathbf{x}\mathbf{x}^T $$


$$ \mathbf{x}\mathbf{y}^T W^{32} = W^{32^T}\mathbf{y}\mathbf{x}^T $$

previous results become $$ 2W^{32^T}(\mathbf{y}^\mu\mathbf{x}^{\mu^T} - W^{32}W^{21}\mathbf{x}^\mu\mathbf{x}^{\mu^T})$$

Then we finally get: $$\Delta W^{21} = \lambda\sum_{\mu=1}^PW^{32^T}(\mathbf{y}^\mu\mathbf{x}^{\mu^T} - W^{32}W^{21}\mathbf{x}^\mu\mathbf{x}^{\mu^T})$$

the same for $\Delta W^{32}$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.