1
$\begingroup$

I would like to ask you why do we need to calculate a derivative of the loss function w.r.t X? It seems like, that for the backpropagation we need to calculate only a derivative w.r.t W.

Can you please explain me where i am making a mistake and ideally provide an example.

$\endgroup$
  • 2
    $\begingroup$ Maybe you should add some more information: where did you find that assertion? Maybe it is just a naming issue(?)... sometimes people assign weird names to normal things and depending on the context the parameters may be named $X$(?)? $\endgroup$ – Fabian Werner Jan 16 at 22:07
3
$\begingroup$

A typical neural network with one hidden layer would be

$$\hat{y} =\sigma(W_2 \sigma(W_1 X + b_1) + b_2)$$

Where $\sigma()$ is a sigmoid function, $W_1$ and $W_2$ a weights for connections between layers, and $b_1$ and $b_2$ are bias vectors. $X$ is a matrix of data with one row per observation and one column per feature. The parameters of the model are $\Theta = (W_1, W_2, b_1, b_2)$. Let's also say that the loss function is $J(\Theta;X) = \frac{1}{2}||y - \hat{y}||^2$ for simplicity.

To fit the model to data, we find the parameters which minimize loss: $\hat{\Theta} = \text{argmin} \, J(\Theta;X)$. One condition which must be true at a local minima is that $\nabla_\Theta J = 0$. That gives us the equations:

$$ \frac{\partial J}{\partial W_1} = 0, \frac{\partial J}{\partial W_2} = 0$$ $$\frac{\partial J}{\partial b_1} = 0, \frac{\partial J}{\partial b_2} = 0$$

The notation used here is from matrix calculus, and we are taking partial derivatives with respect to a matrix (for W) or a vector (for b.) The matrix cookbook may help you understand this notation.

So, to answer the first part of your question: we do not need to take any derivatives with respect to X. Rather, X is held fixed while we take derivatives with respect to other parameters. However, in a typical presentation of the back-prop algorithm, we introduce vectors $z^{(i)}$, $a^{(i)}$ to represent the activation at the $i$-th layer (before and after the non-linear activation function is applied, respectively.) So I take your question to be about $z$ instead of $x$.

A full description of the forward pass of an $L$-layer neural net is given recursively as:

$$ a^{(0)} = X$$ $$a^{(i)} = \sigma(z^{(i)})$$ $$ z^{(i)} = W_i a^{(i-1)} + b_i$$ $$\hat{y} = a^{(L)} $$

For the backwards pass, therefore, we can use the chain rule. For example, let's do $W_1$:

$$ \frac{\partial J}{\partial W_1} = \frac{\partial J}{\partial a^{(2)}} \frac{\partial a^{(2)}}{\partial z^{(2)}} \frac{\partial z^{(2)}}{\partial a^{(1)}} \frac{\partial a^{(1)}}{\partial z^{(1)}} \frac{\partial z^{(1)}}{\partial W_1} $$

Given the forward pass equations given above, it turns out that each of these partial derivatives is straight-forward to calculate. You can verify for yourself that:

$$ \frac{\partial J}{\partial a^{(2)}} = \frac{\partial J}{\partial \hat{y}} = y - \hat{y}$$

$$ \frac{\partial a^{(i)}}{\partial z^{(i)}} = a^{(i)} \circ (1-a^{(i)}) $$

$$ \frac{\partial z^{(i)}}{\partial a^{(i-1)}} = W_i $$

$$ \frac{\partial z^{(i)}}{\partial W_i} = a^{(i-1)} $$

Which are all straight-forward, except perhaps the derivative of the sigmoid were we rely on the slightly non-obvious fact that $\sigma'(x) = \sigma(x) (1-\sigma(x))$.

The justification for introducing the (technically extraneous) concept of $z$ is found in how absolutely obvious and clear it makes the separate steps of the forward and backwards pass. If we had to take $\partial a^{(i)} / \partial a^{(i-1)}$ directly without going through $z$ we would have to think about a non-linear function of a matrix all at once; with $z$ we're able to view it as the element-wise application of a non-linear function (which is Calculus 101) and the partial derivative of a linear expression with respect to a matrix (which is Matrix Calculus 101; see the Cookbook).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.