In a CNN each output location of a feature map is given by the kernel over the previous layer's feature maps. If the receptive field is say 5x5 with 3 channels then there are 5x5x3 weights that are used to find the input to the neuron at this location.
I was wondering what would happen if instead of having a weight matrix of 5x5x3 for this location, if you just had a 5x5x1 matrix and computed an average or max filter over the input feature maps. Has anyone every tried this and is there any research on what the effects would be?
For example, if input x is indexed by i,j,channel then the input to neuron may look like this, where we have 75 different parameters.
z = w1*x111 + w2*x112 + w3*x113 + w4x211 ... + w75*x553
But if we constrained the kernel to reuse the weights for each input channel we might get something like this:
z = MAX(w1*x111 + w2*x211 ..., w1*x112 + w2*x212 + ..., w1*x113 + w2*x213 + ...)
z = w1*x111 + w1*x112 + w1*x113 + w2x211 ... + w25*x553
I'm guessing this may regularize the model and also reduce converge time with less model parameters. In deeper layers where the kernel is 5x5x256, this is now a 6400 weight neuron. This seems like it would lead to over-fitting potential where a 25 weight neuron would generalize better.
I think I read that the LeNet-5 network didn't use all of the channels for each neuron at the next layer, and instead used a combination of previous channels. I believe I read that more recent networks stopped doing this because of computational inefficiencies of sparse matrices? What is the current knowledge around this topic? It seems like fully connecting the kernels to all input layers has the same problems of allowing for over fitting that previous fully connected networks had.
Is there any literature our experiments that have explored this topic deeper?