# Neural Network General Learning Dynamics of Gradient Descent

This might be simple to you but can someone tell me step by step how is matrix form of updating rule of $W^{32}$ and $W^{21}$ derived in this case?

Consider linear three layer neural network model which has input $\mathbf{x}$, first layer to second layer weight matrix $W^{21}$, second layer to third layer weight matrix $W^{32}$. The output $\mathbf{y}$ is defined as: $$\mathbf{y} = W^{32}W^{21}\mathbf{x}$$

consider P examples $\{\mathbf{x}^\mu,\mathbf{y}^\mu\}, \mu=1\cdots P$.

Training is accomplished via gradient descent on the squared error $\sum_{\mu=1}^P||\mathbf{y}^\mu-W^{32}W^{21}\mathbf{x^\mu}||^2$ between the desired feature output, and the network’s feature output. This gradient descent procedure yields the batch learning rule.

$$\Delta W^{21} = \lambda\sum_{\mu=1}^PW^{32^T}(\mathbf{y}^\mu\mathbf{x}^{\mu^T} - W^{32}W^{21}\mathbf{x}^\mu\mathbf{x}^{\mu^T})$$

$$\Delta W^{32} = \lambda\sum_{\mu=1}^P(\mathbf{y}^\mu\mathbf{x}^{\mu^T} - W^{32}W^{21}\mathbf{x}^\mu\mathbf{x}^{\mu^T})W^{21^T}$$

where $\lambda$ is learning rate.

## 2 Answers

General squared error $||y-Wx||^2=(y-Wx)^\top(y-Wx)$. (you missed a square in you sq error formula.)

Expand out your 3 layer error formula using inner products, then refer to matrix calculus formula, for example $\frac{\mathrm{d}y^\top Wx}{\mathrm{d}W} = xy^\top$

According to what jf328 hinted.

Here we expand the function:

$\sum_{\mu=1}^P||\mathbf{y}^\mu-W^{32}W^{21}\mathbf{x^\mu}||^2$

to be

$$$$\sum_{\mu=1}^P(\mathbf{y}^\mu-W^{32}W^{21}\mathbf{x^\mu})^T(\mathbf{y}^\mu-W^{32}W^{21}\mathbf{x^\mu})$$$$ Now consider equation:

$$$$(\mathbf{y}-W^{32}W^{21}\mathbf{x})^T(\mathbf{y}-W^{32}W^{21}\mathbf{x})$$$$

expanding function results to:

$$$$\mathbf{y}^T\mathbf{y} - 2 \mathbf{y}^TW^{32}W^{21}\mathbf{x} + \mathbf{x}^TW^{21} W^{32^T} W^{32}W^{21}\mathbf{x}$$$$

according to matrix differential equation fomula:

$$\frac{\partial y^\top Wx}{\partial W} = xy^\top$$

and

$$\frac{\partial b^TX^TDXc}{\partial X} = D^TXbc^T + DXcb^T$$

$$$$\frac{\partial (\mathbf{y}^T\mathbf{y} - 2 \mathbf{y}^TW^{32}W^{21}\mathbf{x} + \mathbf{x}^TW^{21^T} W^{32^T} W^{32}W^{21}\mathbf{x})}{\partial W^{21}}$$$$

This results to:

$$0 + 2\mathbf{x}\mathbf{y}^TW^{32} + 2 W^{32^T}W^{32}W^{21}\mathbf{x}\mathbf{x}^T$$

as

$$\mathbf{x}\mathbf{y}^T W^{32} = W^{32^T}\mathbf{y}\mathbf{x}^T$$

previous results become $$2W^{32^T}(\mathbf{y}^\mu\mathbf{x}^{\mu^T} - W^{32}W^{21}\mathbf{x}^\mu\mathbf{x}^{\mu^T})$$

Then we finally get: $$\Delta W^{21} = \lambda\sum_{\mu=1}^PW^{32^T}(\mathbf{y}^\mu\mathbf{x}^{\mu^T} - W^{32}W^{21}\mathbf{x}^\mu\mathbf{x}^{\mu^T})$$

the same for $\Delta W^{32}$