Suppose I have an ANN which has one input layer of size $128$, one hidden layer of size $64$ and one output layer of size $10$ for a classification problem. Let's assume we have a training sample of $1000$ vectors of size $128$.
Suppose I want to use the mean squared error loss (I know this is not the best loss for a classification problem but I just want to use it for the sake of understanding) and I want to do vanilla stochastic gradient descent using backpropagation to update the weights.
During an weight update inside an epoch I draw a single vector (input matrix of size $1\times128$) which will have outputs $o_i$ and actual targets $y_i$, $i=1, \dots, 10$. So my loss function is $$L=\frac{1}{10}\displaystyle\sum_{i=1}^{10}(o_i-y_i)^2.$$ Then for updating a weight, say $w_{mn}$, with backpropagation, I need to compute $\frac{\partial L}{\partial w_{mn}}$. Since $\frac{\partial (o_j-y_1)^2}{\partial o_1}=0$ for $j\neq1$, we have $$\frac{\partial L}{\partial w_{mn}}=\frac{\partial L}{\partial o_1}\frac{\partial o_1}{\partial w_{mn}}=\frac{1}{10}\frac{\partial (o_1-y_1)^2}{\partial o_1}\frac{\partial o_1}{\partial w_{mn}}=\dots.$$
So far so good, but now suppose I want to do mini-batch stochastic gradient descent and use backpropagation to update the weights. Suppose the batches are of size $n$.
During an weight update inside an epoch I draw a sample of $n$ vectors of size $128$ (input matrix of size $n\times128$), which will have outputs $o_{ij}$ and targets $y_{ij}$, for $i=1,…,n, j=1, \dots 10$. Again, I want to use the mean squared error loss. Here come my questions:
$(1)$ Will my loss function be $L_i=\frac{1}{10}\displaystyle\sum_{j=1}^{10}(o_{ij}-y_{ij})^2$ for each single vector $i$ in my batch, from which I compute $\frac{\partial L_i}{\partial w_{mn}}$ using backpropagation as above and then I compute the weight update for $w_{mn}$ as $\frac{1}{n} {\displaystyle\sum_{i=1}^{n}} \frac{\partial L_i}{\partial w_{mn}}$? This seems to be the case following backpropagation section of code here for example. What I don't like about this is that it doesn't seem to be right in the way of defining the loss function $L_i$ for each input in the batch, instead of a well defined loss function for the entire batch.
$(2)$ Addressing what I don't like about $(1)$, will the loss function be as suggested here $$ L=\frac{1}{n}\displaystyle\sum_{i=1}^{n}\frac{1}{10}\displaystyle\sum_{j=1}^{10}(o_{ij}-y_{ij})^2?$$
But the when I want to compute the weight update for $w_{mn}$, I can do $$\frac{\partial L}{\partial w_{mn}}=\frac{\partial L}{\partial o_{11}}\frac{\partial o_{11}}{\partial w_{mn}}=\frac{1}{10}\frac{1}{n}\frac{\partial (o_{11}-y_{11})^2}{\partial o_{11}}\frac{\partial o_{11}}{\partial w_{mn}}=\dots,$$ since $\frac{\partial (o_{ij}-y_{ij})^2}{\partial o_{11}}=0$ for $i\neq1$ and $j\neq1$. But this doesn't seem right as this implies only one vector (with $i,j=1$) out of the $n$ in the batch will contribute to the weight update. So what am I missing?
I would like to know if my general understanding is good and in particular what am I missing in case $(2)$.
If anyone could help me, I would be grateful. Thank you.
