I'm quite new to machine learning, and as far as I understand the back-propagation rule is an algorithm the allow to compute the gradient of the cost function defined when training a ANN.

First question, Is my understanding correct?

The second question is related to derivation of the back propagation rule, I've read through few books (the relevant chapters at least, one example is Mitchell, Machine Learning) and watched a couple of videos but I don't find particularly clear the derivations, so I'm trying to propose a different derivation (which I'm pretty sure someone else has worked out).

Now suppose we have $n$ features, and $m$ layer of neurons, each layer has $n_j, 1 \leq j \leq m$ neurons, the last layer is the one that produces output, for a given layer $1 \leq l \leq m$ I define $\omega_{ij}^{l}$ as the weight that defines the arc connecting the neuron $s_{i}^{l-1}, 1 \leq i \leq n_{l-1}$ to $s_j^{l}, 1 \leq j \leq n_{l}$ (In general $s_{i}^{l}$ denotes the $i-th$ activation function at layer $l$.

Please note for simplicity I'm skipping the bias, but I think what I'm about to do wouldn't change.

If $f : \mathbb{R}^{n} \to \mathbb{R}^{n_m}$ is the function define the neural network the cost function is defined as

$$ C\left(\left\{ \omega_{ij}^l \right\}\right) = C = \frac{1}{2} \sum_{k=1}^K \left\lVert f(x_k) - y_k\right\rVert^2 $$

Before computing the gradient I define $s^l$ the column vector of all the activation function (you can assume these are sigmoids), with $z^l$ I denote the column vector of all the inputs at layer $l$, moreover denoting with $\omega_j^l$ the column vector of all weights defining the arcs entering to node $s_i^l$.

Having said this... I think is safe to say in the first place that

$$ \nabla C = \begin{pmatrix} \nabla_{\omega_{1}^1} C \\ \vdots \\ \nabla_{\omega_{n_1}^1} C \\ \vdots \\ \nabla_{\omega_{1}^l} C \\ \vdots \\ \nabla_{\omega_{j}^l} C \\ \vdots \\ \nabla_{\omega_{n_l}^l} C \\ \vdots \\ \nabla_{\omega_{1}^m} C \\ \vdots \\ \nabla_{\omega_{n_m}^m} C \end{pmatrix} $$

Observe also that $\nabla^T_{\omega_j^l} C = J_{\omega_j^l}(C)$, Jacobian matrix wrt $\omega_j^l$. This means I need a general rule for $\nabla_{\omega_j^l} C$, in order to compute the gradient.

First observe

$$ J_{\omega_j^l} (C) = \frac{1}{2} \sum_{k=1}^K J_{\omega_j^l} \left(\left\lVert f(x_k) - y_k \right\rVert^2\right) = \sum_{k=1}^K \left\langle J_{\omega_j^l}(f(x_k)), f(x_k) - y_k \right\rangle, $$

therefore the problem is now computing $J_{\omega_j^l}(f(x_k))$,

Suppose $l = m, 1 \leq j \leq n_m$ then $$ J_{\omega_j^m}(f(x_k)) = J_{\omega_j^m}(s^m) $$ by the chain rule $$ J_{\omega_j^m}(s^m) = J_{z^m}(s^m) J_{\omega_j^m}(z^m) = J_{z^m}(s^m) s^{m-1}, $$

Suppose $l = m - 1, 1 \leq j \leq n_{m-1}$ we have

$$ J_{\omega_j^{m-1}}(f(x_k)) = J_{\omega_j^{m-1}}(s^m) = J_{z^m}(s^m) J_{s^{m-1}}(z^{m}) J_{\omega_j^{m-1}}(s^{m-1}) $$

For generic $l$, I spotted the pattern $$ J_{\omega_j^l}(f(x_k)) = J_{\omega_j^l}(s^m) = \prod_{h=l+1}^m J_{z^h}(s^h) J_{s^{h-1}}(z^h) J_{\omega_j^l}(s^l) $$

Finally for any $l$ the Jacobian $J_{\omega_j^l}(s^l)$ can be easily computed, which will provide a matrix with the derivatives of the sigmoids multiplied by the output of the sigmod of the incident arcs.

Assuming everything is correct to me the idea of the back propagation rule is literally computing $\nabla C$ starting from the last entries towards the first ones. If for fixed $l$ I compute the gradients $\nabla_{\omega_j^l} C$, $1 \leq j \leq n_l$ the computation will involve terms that can be reused (propagated) at layer $l-1$.

Is this interpretation correct?

I'm asking this question because it seems to me that every time people explains how back propagation works they skip a lot of related math.

PS. $s^0$ simply denotes the features, a picture would probably help to understand my notation, but I cannot find a suitable tool to draw an example graph.

If the formulation above works what I like about it is that is indipendent from the activation function, as long as it's differentiable.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.