1
$\begingroup$

This is my network represented in matrices: (a dot represents an arbitrary number) enter image description here Feed-forwarding: (I omitted nesting it all in an activation function for the sake of brevity) enter image description here Backpropagation enter image description here

The question

enter image description here $\partial E/\partial b^3$ should equal a matrix of dimensions: $3 \times 1$, in order to make the subtraction possible by the former $b^3$. The batch size of 5, however, made the dimensions of the $b^3$ matrix equal to $3 \times 5$, which is problematic as you now can't subtract $\partial E/\partial b^3$ because of the dimension mismatch.

What went wrong? Am I supposed to take the average of each row of the $\partial E/\partial b^3$ matrix or perhaps just accumulate each row? Or is it something completely different?

My reflection At the moment I am thinking that what I have done up to now is correct, however I just need to either accumulate or take the average of all the numbers in each row in the matrix $∂E/∂b3$. This way the matrix will be of size $3\times1$ as wanted, and it also makes intuitive sense in the way that I am updating the bias based on an error calculated over an entire batch size, therefor either accumulating or taking the average would make sense. However as I mentioned I am not sure which one it is, or if it is even the right choice.

Any help is highly appreciated!

Improve this question
$\endgroup$
3
  • $\begingroup$ What is you loss function ? It is supposed to be a scalar function so the derivative should be a $3 \times 1$ vector. Also, you never use the bias, are you sure that for example $a^1 = W^1 \cdot X $ instead of $a^1 = W^1 \cdot X + b^1 $ $\endgroup$
    – François Darmon
    Commented Oct 23, 2018 at 9:27
  • $\begingroup$ @user7573566 Would it be incorrect to simply average each row, so that the current $3\times 5$ matrix becomes a $3\times 1$ vector? (I refer to $\partial E/\partial b^3$) $\endgroup$
    – Sebastian Nielsen
    Commented Oct 23, 2018 at 13:05
  • $\begingroup$ @user7573566 My loss function is MSE (mean squared error) $1/M∗(a^3−y)$ where M is the total number of training examples. I left $1/M$ in the above to simplify the example. What scalar function would you recommend I used instead, so the derivative is becomes $3\times1$ vector? $\endgroup$
    – Sebastian Nielsen
    Commented Oct 23, 2018 at 13:45

1 Answer 1

Reset to default
1
$\begingroup$

You use the batched operations in your derivations which makes it more difficult to understand. When you do this, $a^3 = W^3 a^2 + b^3$ is an invalid matrix sum since b is a vector and not a matrix.

You cannot apply usual chain rule

$$ \frac{\partial E}{\partial b} = \frac{\partial E}{\partial a^3}\frac{\partial a^3}{\partial b}$$ because $\frac{\partial a^3}{\partial b}$ is a matrix by vector derivatives. Instead, it is easier to apply chain rule as

$$ \frac{\partial E}{\partial b} = \sum_{i = 1}^3 \frac{\partial E}{\partial a^3_i}\frac{\partial a^3_i}{\partial b}$$ $$ \frac{\partial E}{\partial b} = \sum_{i = 1}^3 2(a^3_i - y_i)$$

Where i is the batch index and $y_i$ the target of the i-th input.

Improve this answer
$\endgroup$
7
  • $\begingroup$ So to clarify, you are saying that I should sum up each row, so the ($3 \times 5$) matrix is reduced to a ($3 \times 1$) vector. Is that correct? $\endgroup$
    – Sebastian Nielsen
    Commented Oct 23, 2018 at 14:16
  • $\begingroup$ Yes, when you compute the chain rule, it appears that you sum each row $\endgroup$
    – François Darmon
    Commented Oct 23, 2018 at 14:20
  • $\begingroup$ Do you happen to have a source that states the same (that you have to sum each row), I don't want to base my knowledge on a single comment (answer) from a stranger online. $\endgroup$
    – Sebastian Nielsen
    Commented Oct 23, 2018 at 14:30
  • $\begingroup$ You can look at the answer, it is simply a chain rule. $\endgroup$
    – François Darmon
    Commented Oct 23, 2018 at 14:39
  • $\begingroup$ Doing the feed-forward phase, the way I added the bias value is not right, as you said yourself: "$a^3=W^3a^2+b^3$ is an invalid matrix sum since b is a vector and not a matrix." But what is the correct way to add the bias then? i.sstatic.net/5YyKp.png $\endgroup$
    – Sebastian Nielsen
    Commented Oct 23, 2018 at 14:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.