So, I was going over the lectures for the Oxford 2015 deep learning course, and in the lectures, they introduce back-propagation as a recursive procedure which involves two key formulas:

  1. The derivative of network error, $E$, wrt layer $\ell$'s inputs, $z^{\ell}$: $$\frac{\partial E}{\partial z^{\ell}} = \frac{\partial E}{\partial z^{\ell+1}} \frac{\partial z^{\ell+1}}{\partial z^{\ell}} = \sum_{j=1}^{J_{\ell+1}} \frac{\partial E}{\partial z^{\ell+1}_{j}} \frac{\partial z^{\ell+1}_{j}}{\partial z^{\ell}}=\sum_{j=1}^{J_{\ell+1}} \delta^{\ell+1}_{j} \frac{\partial z^{\ell+1}_{j}}{\partial z^{\ell}}$$
  2. The derivative of network error, $E$, wrt layer $\ell$'s parameters, $\theta^{\ell}$: $$\frac{\partial E}{\partial \theta^{\ell}} = \frac{\partial E}{\partial z^{\ell+1}} \frac{\partial z^{\ell+1}}{\partial \theta^{\ell}} = \sum_{j=1}^{J_{\ell+1}} \delta^{\ell+1}_{j} \frac{\partial z^{\ell+1}_{j}}{\partial \theta^{\ell}}$$ where

    • $$ z^{\ell+1} = \mathbf{f}(z^{\ell}; \theta^{\ell}) = \left(f_j(z^{\ell}; \theta^{\ell}) : j = 1,\ldots, J_{\ell+1}\right) $$
    • $$\frac{\partial E}{\partial z^{\ell+1}} = \left( \frac{\partial E}{\partial z^{\ell+1}_{j}} : j = 1,\ldots, J_{\ell+1}\right) := \delta^{\ell+1} = \left(\delta^{\ell+1}_{j} : j = 1,\ldots, J_{\ell+1}\right) $$

These formulas aren't my issue -- their derivation seems like a straight forward application of the multivariate Chain Rule -- we can "simplify" both in terms of matrix multiplication:

  • $$ \delta^{\ell} = \delta^{\ell+1} \frac{\partial\mathbf{f}}{\partial z^{\ell}} $$
  • $$ \frac{\partial E}{\partial \theta^{\ell}} = \delta^{\ell+1} \frac{\partial\mathbf{f}}{\partial \theta^{\ell}} $$

However, in this setup, ${\partial\mathbf{f}}/{\partial \theta^{\ell}}$ is a $J_{\ell+1} \times \dim(\theta^{\ell}) $ matrix, and without knowing anything about $\partial f_i / \partial \theta^{\ell}_j$, we cannot do much.

So, let's suppose we have a layer with $I$ input nodes, $J$ output nodes, and where all input nodes are connected to all output nodes, but no inputs nodes are connected (and neither are the output nodes). So, this layer looks like an RBM (for example). Furthermore, let's not suppose anything about relationships between $f_i(z;\theta)$ and $f_j(z;\theta)$ (we'll do that later). Lastly, let's re-index $\theta$ such that $$ \theta = (\theta^{j} : j = 1, \ldots, J) \in \mathbb{R}^{I \times J} $$ and $$ \theta^{j} = (\theta^{j}_{i} : i = 1, \ldots, I) $$ That is $\theta^{j}$ are the weights connecting the input to the $j$-th output node.

We can safely assume that since output nodes are not connected $$ \frac{\partial f_i}{\partial \theta^j} = \mathbf{0} \; \text{ for all $i \neq j$.} $$ Given the above, we notice that ${\partial\mathbf{f}}/{\partial \theta}$ has a very particular structure: $$ \frac{\partial\mathbf{f}}{\partial \theta} = \begin{pmatrix} \partial f_1 / \partial \theta^1 & \mathbf{0} & \cdots & \mathbf{0} \\ \mathbf{0} & \partial f_2 / \partial \theta^2 & \cdots & \mathbf{0} \\ \vdots & \vdots & \ddots & \vdots \\ \mathbf{0} & \mathbf{0} & \cdots & \partial f_J / \partial \theta^J \end{pmatrix} \in \mathbb{R}^{J \times (IJ)} $$

This structure makes writing down $\partial E/ \partial \theta$ dead-simple: $$ \frac{\partial E}{\partial \theta} = \begin{pmatrix} \delta_1 \partial f_1 / \partial \theta^1 & \mathbf{0} & \cdots & \mathbf{0} \\ \mathbf{0} & \delta_2 \partial f_2 / \partial \theta^2 & \cdots & \mathbf{0} \\ \vdots & \vdots & \ddots & \vdots \\ \mathbf{0} & \mathbf{0} & \cdots & \delta_J \partial f_J / \partial \theta^J \end{pmatrix} $$

Here comes my question: If I wanted to save space and store $\partial E/ \partial \theta$ as $$ A := \begin{pmatrix} \delta_1 \partial f_1 / \partial \theta^1 \\ \delta_2 \partial f_2 / \partial \theta^2 \\ \vdots \\ \delta_J \partial f_J / \partial \theta^J \end{pmatrix} $$ is there any way I could use $$ B := \begin{pmatrix} \partial f_1 / \partial \theta^1 \\ \partial f_2 / \partial \theta^2 \\ \vdots \\ \partial f_J / \partial \theta^J \end{pmatrix} $$ and $\delta$ (as shown above) to write $A$ as composition of matrix multiplications and/or vector additions using $B$ and $\delta$?

Special Case: We can see that if $$ (\partial_\theta f =) \frac{\partial f_i}{\partial \theta^i} = \frac{\partial f_j}{\partial \theta^j} \; \text{for all $i$, $j$} $$ then, the outer product, $\delta^T \partial_{\theta} f$ seems to work: $$ \begin{pmatrix} \delta_1 \\ \delta_2 \\ \vdots \\ \delta_J \end{pmatrix} \begin{pmatrix} \partial_{\theta_1} f & \partial_{\theta_2} f & \cdots & \partial_{\theta_I} f \end{pmatrix} = \begin{pmatrix} \delta_1 \partial_{\theta} f \\ \delta_2 \partial_{\theta} f \\ \vdots \\ \delta_J \partial_{\theta} f \end{pmatrix} = A $$ But what about the general case?

  • $\begingroup$ I'm going to move this question to scicomp.stackexchange... $\endgroup$
    – StevieP
    Jun 18, 2015 at 10:00
  • $\begingroup$ If you'd like this question to be migrated to a different site, you just have to flag it for moderator attention & they can take care of it for you. $\endgroup$
    – Sycorax
    Aug 17, 2018 at 2:47


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.