1
$\begingroup$

I have read a few threads on implementing batch training in neural networks. Still I don't understand some specifics of the implementation:

When backpropagating the accumulated error of a given batch $\{X_1, \dots, X_{m}\}$ of size $m$ into a hidden layer, which $X_i$ is used to calculate the error term of the hidden units?

The formula states that for a output unit (hidden units have the same term in their error equation) $j$ the error term used to calculate $\Delta w_{ij}$ is $\delta_j= \varphi'(\sum_{i=1}^{n}x_i w_{ij})(o_j-t_j)$ where $\varphi$ is the used activation function. Should $(x_1, \dots , x_n)$ be an average regarding the batch inputs $\{X_1, \dots, X_{m}\}$?

Also when calculating the average errors: Should I use a simple average by summing up the errors for each component of the output vector individually and then dividing with the batch size to get average error for each output neuron?

Thanks for reading my question.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

In batch training, applications may have slight differences but you sum (or average) the individual gradients calculated for each sample $X_i$. The $x_i$ in the equation for $\delta_j$ denotes the $i$-th entry of the layer's input vector. It's not associated with any specific batch sample. In batch training, you sum the update amounts (or the gradients) calculated for each sample in the batch i.e. $\Delta w_{ij}=\sum_m{\Delta w_{ij}(X_m)}$

Improve this answer
$\endgroup$
3
  • $\begingroup$ Ok, then I think I misunderstood the meaning of batch training. I thought, that you accumulate the error and then calculate the gradient only once. I thought the whole point is less gradient calculation. So what you just said is, that in a batch of size m I still backpropagate m times, but only adjust once using the total sum of deltas, not the average? This confusion is why I asked to which sample (x_1, ... , x_n) belongs. I know that it's a layer input not a sample (at least not for layers_i , i>0), maybe the right formulation would have been to which sample feed forward it belongs. $\endgroup$
    – user266172
    Nov 19, 2019 at 11:57
  • $\begingroup$ It all depends on the cost function. If $J$ has one term, it is SGD, if it has more than one, it's mini-batch or batch. The derivative spreads over the sum. We sum the gradients because $$\frac{\partial J}{\partial w}=\frac{\partial}{\partial w}\left(\sum (y_i-f(X_i))^2\right)=\sum \frac{\partial}{\partial w}\left(y_i-f(X_i)\right)^2$$ $\endgroup$
    – gunes
    Nov 19, 2019 at 12:06
  • $\begingroup$ Now it makes more sense to me. I will tweak around with it. Thank you! $\endgroup$
    – user266172
    Nov 19, 2019 at 12:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.