I recently implemented a neural network, with backpropagation in a fully matrix approach, as described here, where the whole dataset is used for each backprop: http://ufldl.stanford.edu/wiki/index.php/Neural_Network_Vectorization
This worked very quickly, much quicker than one run through of example by example backpropagation, which I will call vectorised here, based more on an iterative style like http://ufldl.stanford.edu/wiki/index.php/Backpropagation_Algorithm
However it took many iterations of the matrix backprop to achieve an 85% accuracy on MNIST, whereas vectorised I got 93% in one epoch. I understand this is because the vectorised approach updates the weights each example, so slowly converges, hence the matrix approach would need a bigger learning rate or much more iterations to achieve the same gradient descent, however, this completely defeated the point of the matrix approach as it ended up taking longer to run it with the required iterations. Note this was not GPU accelerated.
I was wondering if anyone has experience with trying to fully matrix the backprop, and if I am going wrong, or missing something or whether it is in fact best to do example by example because of the updated weights each step.