1
$\begingroup$

Suppose a multi-layer feed-forward neural network, e.g.:

Network Structure

Using matrix form to account for all training samples $(i)$, the forward propagation can be written as follows:

$Z^{[l]}=W^{[l]}A^{[l-1]}+\overline{b}^{[l]}$

$A^{[l]}=g^{[l]}(Z^{[l]})$

where $g^{[l]}$ is the activation function used at layer ${[l]}$.

Let $L$ denote the loss function. For the backpropagation, we want to compute partial derivatives of $L$ with respect $z^{[l](i)}_j$ for all nodes $j$ of the layer $[l]$ and all training examples $(i)$. Many tutorials (e.g. this) call the resulting matrix a Jacobian. I do not understand how this is the case.

In particular, we can view $L$ as a function of the inputs at the layer $[l]$, i.e. $L=L(z^{[l]}_1, z^{[l]}_2,\ldots,z^{[l]}_{n^{[l]}})$. For each training sample, the output of this function is a scalar, whereas the definition of Jacobian requires that the function's output be a vector. So, it seems to me that what we have here (i.e. when we join the derivatives for all the training samples into one matrix form) is not a Jacobian, but a vector of gradients, each computed at a different point. What am I missing?

$\endgroup$

1 Answer 1

3
$\begingroup$

When there is only one function to evaluate, you'll have one row in the Jacobian matrix, i.e. a vector. For completeness, the following quote is from wikipedia:

Suppose $f : ℝ^n → ℝ^m$ is a function such that each of its first-order partial derivatives exist on $ℝ^n$ ... When m = 1, that is when f : $ℝ^n$ → ℝ is a scalar-valued function, the Jacobian matrix reduces to a row vector

$\endgroup$
7
  • $\begingroup$ My question was about joining the derivatives for all the training samples into one matrix form. Then we get a matrix, not just a row vector. How is that matrix a Jacobian? (I have edited the question to clarify) $\endgroup$ Jan 24, 2021 at 8:39
  • $\begingroup$ @AlwaysLearning can you point out where the author of the tutorial you linked thinks the Jacobian has different entries for each training example? $\endgroup$
    – gunes
    Jan 24, 2021 at 14:10
  • $\begingroup$ You are correct. When perusing that tutorial (among many others) I missed that he is talking about stochastic updates... In any case, what notation would you use for the matrix I have in mind? $\endgroup$ Jan 24, 2021 at 15:37
  • $\begingroup$ You don't need to construct such a matrix because gradients for different training examples are averaged (or summed) in backpropagation. $\endgroup$
    – gunes
    Jan 24, 2021 at 15:39
  • $\begingroup$ I would like to derive matrix equations for batch updates of the weights. I would like to do so in matrix form from the beginning to the end, without going first through the derivations for a single training example. This is where the mentioned matrix comes into play. $\endgroup$ Jan 24, 2021 at 16:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.