3
$\begingroup$

I am trying to wrap my head around using batch backprop in a neural network. I have a very code-oriented mind, and I'm trying to figure out whether it's possible to parallelize the full batch backpropagation algorithm.

Say, if the dataset is split up in 100. 100 threads are spawned, and each thread will accumulate the gradients. In the next step, these accumulated gradients are averaged. How would something like this look in pseudocode, not in math equations?

$\endgroup$
2
  • $\begingroup$ For clarity: By full batch, do you mean the version that first feeds forward all of the training data, and then back propagates the summed error? $\endgroup$ Oct 1, 2015 at 1:09
  • $\begingroup$ Yes, it feeds a batch forward, and then back propagates the summed error of the batch. It doesn't feed forward the entire training data at once. $\endgroup$
    – tolgap
    Oct 2, 2015 at 7:31

2 Answers 2

2
$\begingroup$

When using stochastic gradient descent (i.e. batch gradient descent) you update the weights of the network after each batch by the average gradient over all instances in the batch.

So in short no you can not parallelize the batches, but you can however parallelise the process of computing the gradient over the batch, as this gradient is the average over the gradient computed for each instance in the batch.

This last type of parallelisation is usually implemented by structuring the inputs in a matrix and doing matrix multiplications in the forward and backward pass to compute all gradients at the same time. The actual matrix multiplications are then the ones being parallelised/optimised

$\endgroup$
0
$\begingroup$

As it was already told, you can parallelize samples computation in every single serial batch.

I was taught that using multiple number of samples perfectly fits with multicore CPUs, as you can get increased accuracy and convergence rate for free due to parallelization speedup. So make the number of samples in the batch adaptable and equal to the number of CPU cores/threads of the host system.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.