# Neural Network Batch Training

I have read a few threads on implementing batch training in neural networks. Still I don't understand some specifics of the implementation:

When backpropagating the accumulated error of a given batch $$\{X_1, \dots, X_{m}\}$$ of size $$m$$ into a hidden layer, which $$X_i$$ is used to calculate the error term of the hidden units?

The formula states that for a output unit (hidden units have the same term in their error equation) $$j$$ the error term used to calculate $$\Delta w_{ij}$$ is $$\delta_j= \varphi'(\sum_{i=1}^{n}x_i w_{ij})(o_j-t_j)$$ where $$\varphi$$ is the used activation function. Should $$(x_1, \dots , x_n)$$ be an average regarding the batch inputs $$\{X_1, \dots, X_{m}\}$$?

Also when calculating the average errors: Should I use a simple average by summing up the errors for each component of the output vector individually and then dividing with the batch size to get average error for each output neuron?

In batch training, applications may have slight differences but you sum (or average) the individual gradients calculated for each sample $$X_i$$. The $$x_i$$ in the equation for $$\delta_j$$ denotes the $$i$$-th entry of the layer's input vector. It's not associated with any specific batch sample. In batch training, you sum the update amounts (or the gradients) calculated for each sample in the batch i.e. $$\Delta w_{ij}=\sum_m{\Delta w_{ij}(X_m)}$$
• It all depends on the cost function. If $J$ has one term, it is SGD, if it has more than one, it's mini-batch or batch. The derivative spreads over the sum. We sum the gradients because $$\frac{\partial J}{\partial w}=\frac{\partial}{\partial w}\left(\sum (y_i-f(X_i))^2\right)=\sum \frac{\partial}{\partial w}\left(y_i-f(X_i)\right)^2$$ Nov 19, 2019 at 12:06