Strategies for parallelising neural networks When it comes to parallelising a problem, it involves the division of routines and subroutines between a number of nodes, namely; the master node and the slave nodes. Once each of these nodes completes its respective task, all the results from all the nodes are concatenated together to give the final output, hopefully in a shorter time than when using sequential computing methods.
My question is, is there a standard strategy which migrates this method over to the realm of neural networks? For example, my initial reasoning was that what would be "parallelised" in this case would be the feature vector, and each portion of this vector is computed simultaneously, and then concatenated together into one again and the final result will be classified.
Would this strategy make sense or would it be better to parallelise the neurons themselves instead?
 A: What's the distinction between parallelizing the feature vector and parallelizing the neurons? I assume that the neurons of the input layer are features of a training instance. 
Suppose the feature vector of an instance is a 1 by n1 matrix, the weights between the input layer and the first hidden layer is a n1 by n2 matrix, and there are m training instances. Then a feed-forward pass from the input to the hidden layer is just a matrix multiplication, the multiplication of the m by n1 matrix and the n1 by n2 matrix. The training process is made up of many matrix multiplications, so the problem is essentially about parallelizing matrix multiplication. 
In machine learning, researchers more often parallelize training examples instead of parallelize features. There are three types of learning (1) batch learning (2) mini-batch learning (3) stochastic learning, and mini-batch algorithms are often considered for parallelization speedup. The parallelization can be seen as decomposing (m by n1) * (n1 by n2) to many (b by n1) * (n1 by n2) matrix multiplications where b is the batch size. 
In comparison, parallelizing features is less conceptually sensible. Particularly, for deep neural network learning, the features are considered interconnected with each other, and the connections are what the algorithms try to learn. When performing unsupervised pre-training, if features are paralleled, then different parts of features are separated and their connections cannot be learned.  
A: Typically parallel versions do some form of stochastic gradient descent, in which SGD steps for different data instances are spread across nodes (map) and then the partial results are summed up (reduce). This simple strategy can be used to parallelize many algorithms.
This NIPS paper is a very good read on parallelizing machine learning algorithms (including NN) using MapReduce. Quoting from this paper:

By defining a network structure (we use a three layer network with two output neurons classifying the data into two categories), each mapper propagates its set of data through the network. For each training example, the error is back propagated to calculate the partial gradient for each of the weights in the network. The reducer then sums the partial gradient from each mapper and does a batch gradient descent to update the weights of the network.

