4
$\begingroup$

In a CNN each output location of a feature map is given by the kernel over the previous layer's feature maps. If the receptive field is say 5x5 with 3 channels then there are 5x5x3 weights that are used to find the input to the neuron at this location.

I was wondering what would happen if instead of having a weight matrix of 5x5x3 for this location, if you just had a 5x5x1 matrix and computed an average or max filter over the input feature maps. Has anyone every tried this and is there any research on what the effects would be?

For example, if input x is indexed by i,j,channel then the input to neuron may look like this, where we have 75 different parameters.

z = w1*x111 + w2*x112 + w3*x113 + w4x211 ... + w75*x553

But if we constrained the kernel to reuse the weights for each input channel we might get something like this:

z = MAX(w1*x111 + w2*x211 ..., w1*x112 + w2*x212 + ..., w1*x113 + w2*x213 + ...)

or

z = w1*x111 + w1*x112 + w1*x113 + w2x211 ... + w25*x553

I'm guessing this may regularize the model and also reduce converge time with less model parameters. In deeper layers where the kernel is 5x5x256, this is now a 6400 weight neuron. This seems like it would lead to over-fitting potential where a 25 weight neuron would generalize better.

I think I read that the LeNet-5 network didn't use all of the channels for each neuron at the next layer, and instead used a combination of previous channels. I believe I read that more recent networks stopped doing this because of computational inefficiencies of sparse matrices? What is the current knowledge around this topic? It seems like fully connecting the kernels to all input layers has the same problems of allowing for over fitting that previous fully connected networks had.

Is there any literature our experiments that have explored this topic deeper?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

I'll assume you mean for the first layer, because sharing weights across separate channels in subsequent layers would result in all your feature maps for any given layer be exactly the same thing (i.e. same as having one layer).

If you share weights across the three input channels, assuming they're RGB channels, would mean that your model pays the same amount of attention to all channels. This would be equivalent of having a grayscale input, as $gray = \frac{r \, + \, b \, + \, g}{3}$.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Can you explain or give a source to why each layer would learn the same filter if the weights are shared over feature maps ? $\endgroup$
    – Christian Gabor
    Commented Nov 19, 2018 at 22:28
  • $\begingroup$ If you used the same weights, the product of the input with the filter would result in the same values, meaning the same feature map would be generated for a given layer $\endgroup$
    – Javier
    Commented Nov 20, 2018 at 21:46
  • $\begingroup$ I'm asking what would happen if each filter used a subset of inputs and did not learn weights for all inputs in the receptive field. The conv layer may still have a depth of many filters but each of these filters do not have access to all of the previous feature maps. Are we talking about different things? $\endgroup$
    – Christian Gabor
    Commented Nov 21, 2018 at 6:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.