Confusion about feature maps in CNN — why don't they learn the same thing

Question

I have a theoretical question about CNN. As a reference I'm using: http://cs231n.github.io/convolutional-networks/#case

I'm pretty clear on the RELU layer and Pooling Layers. Those are deterministic and easy to figure out. Even the fully connected layers are fairly straightforward -- a gradient descent algorithm is used to reduce minimize the cross entropy. So far so good.

However, I'm confused on the convolution layer. I understand how a convolution works. However, I don't understand is why different convolutions are learned for different feature maps.

For example, suppose I start with a 32 x 32 x 3 image. Suppose I use a stride of 1 and a pad of 1. That means my first neuron is connected to 32x32x3=3072 pixels (plus +1 for a bias). Specifically it connects to [1:32, 1:32, 1:3]. My second neuron connects [2:33, 2:33, 1:3] and so forth. So I ask it to train and it estimates 3073 total parameters. Great.

Now suppose I want a second feature map. The convolutions are calculated in parallel as they only take the original receiving image. So the first neuron again takes pixels [1:32, 1:32, 1:3]. The second neuron again takes pixels [2:33, 2:33, 1:3] etc. Again it estimates 3073 parameters.

Why won't it learn the exact same values for the 3073 parameters as it's just doing gradient descent?

In other words why might one feature learn parameters that activate on a "squiggly curve" and one feature learn parameters that activate a "90 degree angle". They both start with the same image and they both do gradient descent!

Thanks in advance for your help.

Answer 1

This is exactly the same reasoning to why parallel neurons in general not learn the same thing. By doing random initialization the neurons are by construction different and as the optimization progresses the neurons have no benefit of reproducing other neuron. So the gradients are not the same otherwise you are correct. Check out: http://cs231n.github.io/neural-networks-2/#init

Lets start with what we should not do. Note that we do not know what the final value of every weight should be in the trained network, but with proper data normalization it is reasonable to assume that approximately half of the weights will be positive and half of them will be negative. A reasonable-sounding idea then might be to set all the initial weights to zero, which we expect to be the “best guess” in expectation. This turns out to be a mistake, because if every neuron in the network computes the same output, then they will also all compute the same gradients during backpropagation and undergo the exact same parameter updates. In other words, there is no source of asymmetry between neurons if their weights are initialized to be the same.