I have a theoretical question about CNN. As a reference I'm using: http://cs231n.github.io/convolutional-networks/#case

I'm pretty clear on the RELU layer and Pooling Layers. Those are deterministic and easy to figure out. Even the fully connected layers are fairly straightforward -- a gradient descent algorithm is used to reduce minimize the cross entropy. So far so good.

However, I'm confused on the convolution layer. I understand how a convolution works. However, I don't understand is why different convolutions are learned for different feature maps.

For example, suppose I start with a 32 x 32 x 3 image. Suppose I use a stride of 1 and a pad of 1. That means my first neuron is connected to 32x32x3=3072 pixels (plus +1 for a bias). Specifically it connects to [1:32, 1:32, 1:3]. My second neuron connects [2:33, 2:33, 1:3] and so forth. So I ask it to train and it estimates 3073 total parameters. Great.

Now suppose I want a second feature map. The convolutions are calculated in parallel as they only take the original receiving image. So the first neuron again takes pixels [1:32, 1:32, 1:3]. The second neuron again takes pixels [2:33, 2:33, 1:3] etc. Again it estimates 3073 parameters.

Why won't it learn the exact same values for the 3073 parameters as it's just doing gradient descent?

In other words why might one feature learn parameters that activate on a "squiggly curve" and one feature learn parameters that activate a "90 degree angle". They both start with the same image and they both do gradient descent!

Thanks in advance for your help.


This is exactly the same reasoning to why parallel neurons in general not learn the same thing. By doing random initialization the neurons are by construction different and as the optimization progresses the neurons have no benefit of reproducing other neuron. So the gradients are not the same otherwise you are correct. Check out: http://cs231n.github.io/neural-networks-2/#init

Lets start with what we should not do. Note that we do not know what the final value of every weight should be in the trained network, but with proper data normalization it is reasonable to assume that approximately half of the weights will be positive and half of them will be negative. A reasonable-sounding idea then might be to set all the initial weights to zero, which we expect to be the “best guess” in expectation. This turns out to be a mistake, because if every neuron in the network computes the same output, then they will also all compute the same gradients during backpropagation and undergo the exact same parameter updates. In other words, there is no source of asymmetry between neurons if their weights are initialized to be the same.

  • 1
    $\begingroup$ I thought that might be it but I'm not completely convinced. Say my image has a very strong feature -- eg if there is a big horizontal black line it means a positive class (and let's say it's just 2 classes). Won't many feature maps of the CNN (if not all of them at that depth) then simply learn that big black line? $\endgroup$ – user1357015 Jul 20 '17 at 6:48
  • 1
    $\begingroup$ The gradient of well classified samples are smaller than the gradients from poorly classified samples. When the network have learned that black lines means positive class the samples which obeys this pattern will contribute very little to the gradient since they are well classified. So the gradient signal will be more about modeling the residual. So not modeling the black lines. If there are no residual sure you are correct. But then it is not a very interesting problem. $\endgroup$ – Peter Mølgaard Pallesen Jul 20 '17 at 7:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.