What differentiates one feature map from another in CNN I understand in a convolution neural net that you may have several feature maps in the same layer, for instance one map detects curly loops for some letter and another detects straight lines. Here my main question...
Since we do not know features ahead of time, hence the training, what makes a feature map pick out a particular feature instead of doing some stochastic random walk? Since we are testing against the entire grid, it seems like letter features in one area of the grid will get washed out from noise in another area.
 A: A short answer would be: During training you optimize some loss function wrt to the parameters of the network, eg the convolutional features. Doing that, it turns out that edge or curvy features simply lead to a lower error than random features. 
Also note that you usually have fully connected layers higher up in the hierarchy. That way, your network can reason about the location in which some features are strongly present. 
A: Each kernel learns different representation of the input data if their initial points are different. That is why they are randomly initialized. Kernels may learn the same representation if input has lack of structure. Also some restrictions like weight decay may smooth the cost space, so even differently initialized weights can converge to the same local minimum ( because there is not much local minima!). I've encountered this while working with ECG signals with NN. More than 100 different weight groups (each group connected to one hidden unit) converged to only 2 different weight groups. In your case you can think these weight groups as kernels of Convnet. But I don't think kernels converge to the same group of weights at natural images, speech recognition etc., because it is highly structured. The cost space is non-convex and complicated.
