30
$\begingroup$

[This question was also posed at stack overflow]

The question in short

I'm studying convolutional neural networks, and I believe that these networks do not treat every input neuron (pixel/parameter) equivalently. Imagine we have a deep network (many layers) that applies convolution on some input image. The neurons in the "middle" of the image have many unique pathways to many deeper layer neurons, which means that a small variation in the middle neurons has a strong effect on the output. However, the neurons at the edge of the image have only $1$ way (or, depending on the exact implementation, of the order of $1$) pathways in which their information flows through the graph. It seems that these are "under-represented".

I am concerned about this, as this discrimination of edge neurons scales exponentially with the depth (number of layers) of the network. Even adding a max-pooling layer won't halt the exponential increase, only a full connection brings all neurons on equal footing. I'm not convinced that my reasoning is correct, though, so my questions are:

  • Am I right that this effect takes place in deep convolutional networks?
  • Is there any theory about this, has it ever been mentioned in literature?
  • Are there ways to overcome this effect?

Because I'm not sure if this gives sufficient information, I'll elaborate a bit more about the problem statement, and why I believe this is a concern.

More detailed explanation

Imagine we have a deep neural network that takes an image as input. Assume we apply a convolutional filter of $64\times 64$ pixel over the image, where we shift the convolution window by $4$ pixels each time. This means that every neuron in the input sends it's activation to $16 \times 16 = 265$ neurons in layer $2$. Each of these neurons might send their activation to another $265$, such that our topmost neuron is represented in $265^2$ output neurons, and so on.

This is, however, not true for neurons on the edges: these might be represented in only a small number of convolution windows, thus causing them to activate (of the order of) only $1$ neuron in the next layer. Using tricks such as mirroring along the edges won't help this: the second-layer-neurons that will be projected to are still at the edges, which means that that the second-layer-neurons will be underrepresented (thus limiting the importance of our edge neurons as well). As can be seen, this discrepancy scales exponentially with the number of layers.

I have created an image to visualize the problem, which can be found here (I'm not allowed to include images in the post itself). This network has a convolution window of size $3$. The numbers next to neurons indicate the number of pathways down to the deepest neuron. The image is reminiscent of Pascal's Triangle.

https://www.dropbox.com/s/7rbwv7z14j4h0jr/deep_conv_problem_stackxchange.png?dl=0

Why is this a problem?

This effect doesn't seem to be a problem at first sight: In principle, the weights should automatically adjust in such a way that the network does it's job. Moreover, the edges of an image are not that important anyway in image recognition. This effect might not be noticeable in everyday image recognition tests, but it still concerns me because of two reasons: 1. generalization to other applications, and 2. problems arising in the case of very deep networks.

1. There might be other applications, like speech or sound recognition, where it is not true that the middle-most neurons are the most important. Applying convolution is often done in this field, but I haven't been able to find any papers that mention the effect that I'm concerned with.

2. Very deep networks will notice an exponentially bad effect of the discrimination of boundary neurons, which means that central neurons can be overrepresented by multiple order of magnitude (imagine we have $10$ layers such that the above example would give $265^{10}$ ways the central neurons can project their information). As one increases the number of layers, one is bound to hit a limit where weights cannot feasibly compensate for this effect.

Now imagine we perturb all neurons by a small amount. The central neurons will cause the output to change more strongly by several orders of magnitude, compared to the edge neurons. I believe that for general applications, and for very deep networks, ways around my problem should be found?

$\endgroup$
  • 8
    $\begingroup$ I can't fully answer your question, but I found this paper: cs.utoronto.ca/~kriz/conv-cifar10-aug2010.pdf which addresses your problem. They talk about different solutions, 1) padding the edges of the inputs with zeros, 2) adding in randomly globally connected components, or 3) fixing and forward propagating the edges so no edge info lost. I was recently curious about the same problem and found your question. I'm still wading through the details of that paper. Hope it helps. $\endgroup$ – nfmcclure Feb 4 '15 at 22:30
  • $\begingroup$ Thanks, this was exactly what I was looking for! Impressive that this is noticeable with as little as 2 layers. For those TL;DR'ing the whole paper: Using a combination of 1) (zero padding) and 2) (random global connections) was found to be the best remedy. $\endgroup$ – Koen Feb 6 '15 at 11:33
  • $\begingroup$ Related: area51.stackexchange.com/proposals/93481/… $\endgroup$ – kenorb Jun 27 '16 at 18:55
  • $\begingroup$ I am curious, is it not enough using a non overlapping offset? So in your example you have 64x64 patches and you move your input of 64 pixels every time while applying convolution. (your 'shift' is my 'offset'). Which is, i guess, just the same as doing zero padding? $\endgroup$ – Renthal Sep 19 '16 at 20:01
  • 1
    $\begingroup$ Sorry for not responding for a while, I am not working on NN anymore. However, I think I could answer the questions, although I realize my greater goal was to better understand what is going on (which I still don't). 1) This effect seems to take place indeed. 2) The paper linked above describes it and 3) also explores ways to overcome it. To Renthal: Nonoverlapping patches (choosing offset = conv. size) should overcome my problem indeed, but people often find overlapping patches to work better. To EngrStudent, Sorry, I'm not familiar with Bootstrap Resampling. $\endgroup$ – Koen Jan 16 '17 at 13:58
4
$\begingroup$

Sparse representations are expected in hierarchical models. Possibly, what you are discovering is a problem intrinsic to the hierarchical structure of deep learning models. You will find quite a few scientific papers on "sparse representations", especially in memory research.

I think you would benefit from reading about "receptor fields" in visual cortex. Not only are there ON and OFF cells in the mammal brain, but also RF cells that fire both during ON and OFF. Perhaps the edge/sparsity problem could be circumvented by updating the model to reflect current neuroscience on vision, especially in animal models.

$\endgroup$
3
$\begingroup$

You're right that this an issue if the convolution operates only on the image pixels, but the problem disappears if you zero-pad the images (as is generally recommended). This ensures that the convolution will apply the filter the same number of times to each pixel.

$\endgroup$
  • 1
    $\begingroup$ I am not convinced yet. Zero-padding will only create more output-pixels around the "center" pixels one finds without padding. The values of the "center" pixels are exactly the same, no matter what zero-padding one uses. $\endgroup$ – Koen May 2 '17 at 11:43
  • 1
    $\begingroup$ If the convolutional window is of size $n \times n$, then by padding all of the edges with $n-1$ many zeros, you will guarantee that the convolution will act on the edge pixels just as many times as the inner pixels (assuming you do this at every convolutional layer in the network). In practice though, not having such aggressive padding, and instead downweighting the importance of the edge pixels in fine, since the important information is much more likely to be located near the center of the image. $\endgroup$ – jon_simon May 2 '17 at 18:12
  • $\begingroup$ The convolution visualization half-way down this page may help to convey the intuition: cs231n.github.io/convolutional-networks $\endgroup$ – jon_simon May 2 '17 at 18:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.