I have a theoretical question about CNN. As a reference I'm using: http://cs231n.github.io/convolutional-networks/#case
I'm pretty clear on the RELU layer and Pooling Layers. Those are deterministic and easy to figure out. Even the fully connected layers are fairly straightforward -- a gradient descent algorithm is used to reduce minimize the cross entropy. So far so good.
However, I'm confused on the convolution layer. I understand how a convolution works. However, I don't understand is why different convolutions are learned for different feature maps.
For example, suppose I start with a 32 x 32 x 3 image. Suppose I use a stride of 1 and a pad of 1. That means my first neuron is connected to 32x32x3=3072 pixels (plus +1 for a bias). Specifically it connects to [1:32, 1:32, 1:3]. My second neuron connects [2:33, 2:33, 1:3] and so forth. So I ask it to train and it estimates 3073 total parameters. Great.
Now suppose I want a second feature map. The convolutions are calculated in parallel as they only take the original receiving image. So the first neuron again takes pixels [1:32, 1:32, 1:3]. The second neuron again takes pixels [2:33, 2:33, 1:3] etc. Again it estimates 3073 parameters.
Why won't it learn the exact same values for the 3073 parameters as it's just doing gradient descent?
In other words why might one feature learn parameters that activate on a "squiggly curve" and one feature learn parameters that activate a "90 degree angle". They both start with the same image and they both do gradient descent!
Thanks in advance for your help.