# Backpropagation in mini-batch stochastic gradient descent with mean squared error loss

Suppose I have an ANN which has one input layer of size $$128$$, one hidden layer of size $$64$$ and one output layer of size $$10$$ for a classification problem. Let's assume we have a training sample of $$1000$$ vectors of size $$128$$.

Suppose I want to use the mean squared error loss (I know this is not the best loss for a classification problem but I just want to use it for the sake of understanding) and I want to do vanilla stochastic gradient descent using backpropagation to update the weights.

During an weight update inside an epoch I draw a single vector (input matrix of size $$1\times128$$) which will have outputs $$o_i$$ and actual targets $$y_i$$, $$i=1, \dots, 10$$. So my loss function is $$L=\frac{1}{10}\displaystyle\sum_{i=1}^{10}(o_i-y_i)^2.$$ Then for updating a weight, say $$w_{mn}$$, with backpropagation, I need to compute $$\frac{\partial L}{\partial w_{mn}}$$. Since $$\frac{\partial (o_j-y_1)^2}{\partial o_1}=0$$ for $$j\neq1$$, we have $$\frac{\partial L}{\partial w_{mn}}=\frac{\partial L}{\partial o_1}\frac{\partial o_1}{\partial w_{mn}}=\frac{1}{10}\frac{\partial (o_1-y_1)^2}{\partial o_1}\frac{\partial o_1}{\partial w_{mn}}=\dots.$$
So far so good, but now suppose I want to do mini-batch stochastic gradient descent and use backpropagation to update the weights. Suppose the batches are of size $$n$$.

During an weight update inside an epoch I draw a sample of $$n$$ vectors of size $$128$$ (input matrix of size $$n\times128$$), which will have outputs $$o_{ij}$$ and targets $$y_{ij}$$, for $$i=1,…,n, j=1, \dots 10$$. Again, I want to use the mean squared error loss. Here come my questions:

$$(1)$$ Will my loss function be $$L_i=\frac{1}{10}\displaystyle\sum_{j=1}^{10}(o_{ij}-y_{ij})^2$$ for each single vector $$i$$ in my batch, from which I compute $$\frac{\partial L_i}{\partial w_{mn}}$$ using backpropagation as above and then I compute the weight update for $$w_{mn}$$ as $$\frac{1}{n} {\displaystyle\sum_{i=1}^{n}} \frac{\partial L_i}{\partial w_{mn}}$$? This seems to be the case following backpropagation section of code here for example. What I don't like about this is that it doesn't seem to be right in the way of defining the loss function $$L_i$$ for each input in the batch, instead of a well defined loss function for the entire batch.

$$(2)$$ Addressing what I don't like about $$(1)$$, will the loss function be as suggested here $$L=\frac{1}{n}\displaystyle\sum_{i=1}^{n}\frac{1}{10}\displaystyle\sum_{j=1}^{10}(o_{ij}-y_{ij})^2?$$

But the when I want to compute the weight update for $$w_{mn}$$, I can do $$\frac{\partial L}{\partial w_{mn}}=\frac{\partial L}{\partial o_{11}}\frac{\partial o_{11}}{\partial w_{mn}}=\frac{1}{10}\frac{1}{n}\frac{\partial (o_{11}-y_{11})^2}{\partial o_{11}}\frac{\partial o_{11}}{\partial w_{mn}}=\dots,$$ since $$\frac{\partial (o_{ij}-y_{ij})^2}{\partial o_{11}}=0$$ for $$i\neq1$$ and $$j\neq1$$. But this doesn't seem right as this implies only one vector (with $$i,j=1$$) out of the $$n$$ in the batch will contribute to the weight update. So what am I missing?

I would like to know if my general understanding is good and in particular what am I missing in case $$(2)$$.

If anyone could help me, I would be grateful. Thank you.

(1) This is correct because the total loss function is $$L=\frac{1}{n}\sum_{i=1}^n L_i$$ This is even sometimes defined as the sum of losses, not the average. Differentiation simply spreads over the sum/average.

(2) You have a mistake in differentiation (in single sample case as well). If you have a function $$f(x,y)$$ and you want to differentiate $$f$$ wrt another variable $$t$$, the chain rule is as follows:

$$\frac{\partial f}{\partial t}=\frac{\partial f}{\partial x}\frac{\partial x}{\partial t}+\frac{\partial f}{\partial y}\frac{\partial y}{\partial t}$$

Using this, your chain rule for the single sample case should have been written as follows:

$$\frac{\partial L}{\partial w_{mn}}=\sum_{i=1}^n\frac{\partial L}{\partial o_i}\frac{\partial o_i}{\partial w_{mn}}=\sum_{i=1}^n \frac{1}{10}\frac{\partial (o_i-y_i)^2}{\partial o_i}\frac{\partial o_i}{\partial w_{mn}}$$

because $$L$$ is a function of $$o_i$$, $$i\in \{1,2...10\}$$. So, you can't get away with only $$o_1$$.

If you reflect this into the mini-batch formula, you can't get away with only $$o_{11}$$ either. The differentiation should include all of them.

• Thank you very much! So $(1)$ and $(2)$ are actually equivalent, or even the same thing. I guess form $(1)$ is preferred in implementations because it's somewhat easier to see what's going on in terms of differentiation. Jul 23, 2022 at 20:25
• Sorry to insist on this, I completely agree with what you said about the use of chain rule, but for eample looking here cs.cornell.edu/courses/cs5740/2016sp/resources/backprop.pdf at equations $(8)$ and $(10)$ on pages 5-6, they seem to apply the wrong chain rule formula. Is it a mistake in the notes? Jul 24, 2022 at 14:05
• It's the same principle. There, you have $y_j=f(x_j)$ and $x_j=\sum w_{kj}y'_k$ in Figure 3. Some "cross"-derivatives are $0$, that's why it's more simpler. For example, derivative of $y_j$ wrt $x_i$ is $0$ for i != j. That simplifies the partial derivative sum. But, you can't choose an arbitrary output like in your calculations. Jul 28, 2022 at 7:57