Skip to main content
edited body
Source Link
Enes
  • 121
  • 5

$\frac{\partial \mathcal{L}}{\partial W^{[2]}}$ must be 2x3 as just like dimensions of $ W^{[2]}$.

I suggest you to use the backprop formulas (and notation) given in Nielsen's book. When the networks gets bigger it is ease to follow

BP from Nielsen

According to that

\begin{align*} \delta^3 &=a^{[3]}-y \\ \delta^2 &= ((W^{[3]^{T}} (a^{[3]}-y)) \odot g'(z^{[2]})) \\ \frac{\partial \mathcal{L}}{\partial w^{[2]}_{kj}} &= a^{[1]}_k \cdot \delta_j^2 \end{align*}\begin{align*} \delta^3 &=a^{[3]}-y \\ \delta^2 &= ((W^{[3]^{T}} (a^{[3]}-y)) \odot g'(z^{[2]})) \\ \frac{\partial \mathcal{L}}{\partial w^{[2]}_{jk}} &= a^{[1]}_k \cdot \delta_j^2 \end{align*}

And by going one more step:

\begin{align*} \delta^1 &= ((W^{[2]^{T}} \delta^2 ) \odot g'(z^{[1]})) \\ \frac{\partial \mathcal{L}}{\partial w^{[1]}_{kj}} &= x^{(i)}_k \cdot \delta_j^1 \end{align*}\begin{align*} \delta^1 &= ((W^{[2]^{T}} \delta^2 ) \odot g'(z^{[1]})) \\ \frac{\partial \mathcal{L}}{\partial w^{[1]}_{jk}} &= x^{(i)}_k \cdot \delta_j^1 \end{align*}

where $ \delta^1 \in \mathbb{R}^{3\times 1}$

I hope at least for other people it will be usefull

$\frac{\partial \mathcal{L}}{\partial W^{[2]}}$ must be 2x3 as just like dimensions of $ W^{[2]}$.

I suggest you to use the backprop formulas (and notation) given in Nielsen's book. When the networks gets bigger it is ease to follow

BP from Nielsen

According to that

\begin{align*} \delta^3 &=a^{[3]}-y \\ \delta^2 &= ((W^{[3]^{T}} (a^{[3]}-y)) \odot g'(z^{[2]})) \\ \frac{\partial \mathcal{L}}{\partial w^{[2]}_{kj}} &= a^{[1]}_k \cdot \delta_j^2 \end{align*}

And by going one more step:

\begin{align*} \delta^1 &= ((W^{[2]^{T}} \delta^2 ) \odot g'(z^{[1]})) \\ \frac{\partial \mathcal{L}}{\partial w^{[1]}_{kj}} &= x^{(i)}_k \cdot \delta_j^1 \end{align*}

where $ \delta^1 \in \mathbb{R}^{3\times 1}$

I hope at least for other people it will be usefull

$\frac{\partial \mathcal{L}}{\partial W^{[2]}}$ must be 2x3 as just like dimensions of $ W^{[2]}$.

I suggest you to use the backprop formulas (and notation) given in Nielsen's book. When the networks gets bigger it is ease to follow

BP from Nielsen

According to that

\begin{align*} \delta^3 &=a^{[3]}-y \\ \delta^2 &= ((W^{[3]^{T}} (a^{[3]}-y)) \odot g'(z^{[2]})) \\ \frac{\partial \mathcal{L}}{\partial w^{[2]}_{jk}} &= a^{[1]}_k \cdot \delta_j^2 \end{align*}

And by going one more step:

\begin{align*} \delta^1 &= ((W^{[2]^{T}} \delta^2 ) \odot g'(z^{[1]})) \\ \frac{\partial \mathcal{L}}{\partial w^{[1]}_{jk}} &= x^{(i)}_k \cdot \delta_j^1 \end{align*}

where $ \delta^1 \in \mathbb{R}^{3\times 1}$

I hope at least for other people it will be usefull

Source Link
Enes
  • 121
  • 5

$\frac{\partial \mathcal{L}}{\partial W^{[2]}}$ must be 2x3 as just like dimensions of $ W^{[2]}$.

I suggest you to use the backprop formulas (and notation) given in Nielsen's book. When the networks gets bigger it is ease to follow

BP from Nielsen

According to that

\begin{align*} \delta^3 &=a^{[3]}-y \\ \delta^2 &= ((W^{[3]^{T}} (a^{[3]}-y)) \odot g'(z^{[2]})) \\ \frac{\partial \mathcal{L}}{\partial w^{[2]}_{kj}} &= a^{[1]}_k \cdot \delta_j^2 \end{align*}

And by going one more step:

\begin{align*} \delta^1 &= ((W^{[2]^{T}} \delta^2 ) \odot g'(z^{[1]})) \\ \frac{\partial \mathcal{L}}{\partial w^{[1]}_{kj}} &= x^{(i)}_k \cdot \delta_j^1 \end{align*}

where $ \delta^1 \in \mathbb{R}^{3\times 1}$

I hope at least for other people it will be usefull