4
$\begingroup$

Suppose I have an ANN which has one input layer of size $128$, one hidden layer of size $64$ and one output layer of size $10$ for a classification problem. Let's assume we have a training sample of $1000$ vectors of size $128$.

Suppose I want to use the mean squared error loss (I know this is not the best loss for a classification problem but I just want to use it for the sake of understanding) and I want to do vanilla stochastic gradient descent using backpropagation to update the weights.

During an weight update inside an epoch I draw a single vector (input matrix of size $1\times128$) which will have outputs $o_i$ and actual targets $y_i$, $i=1, \dots, 10$. So my loss function is $$L=\frac{1}{10}\displaystyle\sum_{i=1}^{10}(o_i-y_i)^2.$$ Then for updating a weight, say $w_{mn}$, with backpropagation, I need to compute $\frac{\partial L}{\partial w_{mn}}$. Since $\frac{\partial (o_j-y_1)^2}{\partial o_1}=0$ for $j\neq1$, we have $$\frac{\partial L}{\partial w_{mn}}=\frac{\partial L}{\partial o_1}\frac{\partial o_1}{\partial w_{mn}}=\frac{1}{10}\frac{\partial (o_1-y_1)^2}{\partial o_1}\frac{\partial o_1}{\partial w_{mn}}=\dots.$$
So far so good, but now suppose I want to do mini-batch stochastic gradient descent and use backpropagation to update the weights. Suppose the batches are of size $n$.

During an weight update inside an epoch I draw a sample of $n$ vectors of size $128$ (input matrix of size $n\times128$), which will have outputs $o_{ij}$ and targets $y_{ij}$, for $i=1,…,n, j=1, \dots 10$. Again, I want to use the mean squared error loss. Here come my questions:

$(1)$ Will my loss function be $L_i=\frac{1}{10}\displaystyle\sum_{j=1}^{10}(o_{ij}-y_{ij})^2$ for each single vector $i$ in my batch, from which I compute $\frac{\partial L_i}{\partial w_{mn}}$ using backpropagation as above and then I compute the weight update for $w_{mn}$ as $\frac{1}{n} {\displaystyle\sum_{i=1}^{n}} \frac{\partial L_i}{\partial w_{mn}}$? This seems to be the case following backpropagation section of code here for example. What I don't like about this is that it doesn't seem to be right in the way of defining the loss function $L_i$ for each input in the batch, instead of a well defined loss function for the entire batch.

$(2)$ Addressing what I don't like about $(1)$, will the loss function be as suggested here $$ L=\frac{1}{n}\displaystyle\sum_{i=1}^{n}\frac{1}{10}\displaystyle\sum_{j=1}^{10}(o_{ij}-y_{ij})^2?$$

But the when I want to compute the weight update for $w_{mn}$, I can do $$\frac{\partial L}{\partial w_{mn}}=\frac{\partial L}{\partial o_{11}}\frac{\partial o_{11}}{\partial w_{mn}}=\frac{1}{10}\frac{1}{n}\frac{\partial (o_{11}-y_{11})^2}{\partial o_{11}}\frac{\partial o_{11}}{\partial w_{mn}}=\dots,$$ since $\frac{\partial (o_{ij}-y_{ij})^2}{\partial o_{11}}=0$ for $i\neq1$ and $j\neq1$. But this doesn't seem right as this implies only one vector (with $i,j=1$) out of the $n$ in the batch will contribute to the weight update. So what am I missing?

I would like to know if my general understanding is good and in particular what am I missing in case $(2)$.

If anyone could help me, I would be grateful. Thank you.

$\endgroup$

1 Answer 1

2
$\begingroup$

(1) This is correct because the total loss function is $$L=\frac{1}{n}\sum_{i=1}^n L_i$$ This is even sometimes defined as the sum of losses, not the average. Differentiation simply spreads over the sum/average.

(2) You have a mistake in differentiation (in single sample case as well). If you have a function $f(x,y)$ and you want to differentiate $f$ wrt another variable $t$, the chain rule is as follows:

$$\frac{\partial f}{\partial t}=\frac{\partial f}{\partial x}\frac{\partial x}{\partial t}+\frac{\partial f}{\partial y}\frac{\partial y}{\partial t}$$

Using this, your chain rule for the single sample case should have been written as follows:

$$\frac{\partial L}{\partial w_{mn}}=\sum_{i=1}^n\frac{\partial L}{\partial o_i}\frac{\partial o_i}{\partial w_{mn}}=\sum_{i=1}^n \frac{1}{10}\frac{\partial (o_i-y_i)^2}{\partial o_i}\frac{\partial o_i}{\partial w_{mn}}$$

because $L$ is a function of $o_i$, $i\in \{1,2...10\}$. So, you can't get away with only $o_1$.

If you reflect this into the mini-batch formula, you can't get away with only $o_{11}$ either. The differentiation should include all of them.

$\endgroup$
3
  • $\begingroup$ Thank you very much! So $(1)$ and $(2)$ are actually equivalent, or even the same thing. I guess form $(1)$ is preferred in implementations because it's somewhat easier to see what's going on in terms of differentiation. $\endgroup$
    – harlem
    Commented Jul 23, 2022 at 20:25
  • $\begingroup$ Sorry to insist on this, I completely agree with what you said about the use of chain rule, but for eample looking here cs.cornell.edu/courses/cs5740/2016sp/resources/backprop.pdf at equations $(8)$ and $(10)$ on pages 5-6, they seem to apply the wrong chain rule formula. Is it a mistake in the notes? $\endgroup$
    – harlem
    Commented Jul 24, 2022 at 14:05
  • 1
    $\begingroup$ It's the same principle. There, you have $y_j=f(x_j)$ and $x_j=\sum w_{kj}y'_k$ in Figure 3. Some "cross"-derivatives are $0$, that's why it's more simpler. For example, derivative of $y_j$ wrt $x_i$ is $0$ for i != j. That simplifies the partial derivative sum. But, you can't choose an arbitrary output like in your calculations. $\endgroup$
    – gunes
    Commented Jul 28, 2022 at 7:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.