Full batch backpropagation implementation I am trying to wrap my head around using batch backprop in a neural network. I have a very code-oriented mind, and I'm trying to figure out whether it's possible to parallelize the full batch backpropagation algorithm.
Say, if the dataset is split up in 100. 100 threads are spawned, and each thread will accumulate the gradients. In the next step, these accumulated gradients are averaged. How would something like this look in pseudocode, not in math equations?
 A: When using stochastic gradient descent (i.e. batch gradient descent) you update the weights of the network after each batch by the average gradient over all instances in the batch.
So in short no you can not parallelize the batches, but you can however parallelise the process of computing the gradient over the batch, as this gradient is the average over the gradient computed for each instance in the batch.
This last type of parallelisation is usually implemented by structuring the inputs in a matrix and doing matrix multiplications in the forward and backward pass to compute all gradients at the same time. The actual matrix multiplications are then the ones being parallelised/optimised 
A: As it was already told, you can parallelize samples computation in every single serial batch.
I was taught that using multiple number of samples perfectly fits with multicore CPUs, as you can get increased accuracy and convergence rate for free due to parallelization speedup. So make the number of samples in the batch adaptable and equal to the number of CPU cores/threads of the host system.
