Convolutional neural networks (CNNs) do regularization (of sorts) by forcing some weights to be dropped and others to be zero. Borrowing some drawings from another post of mine...
Apply the filter to the upper left 2x2 array.
Apply the filter to the upper right 2x2 array.
Apply the filter to the bottom left 2x2 array.
Apply the filter to the bottom right 2x2 array.
Here is the entire layer, with the 3x3 input image mapping to four neurons for the four positions in the image where convolution occurs.
If we see the next layer as the output, then the entire equation would be something like this (I am leaving out the biases for the sake of clarity, but they should be there to be $100\%$ correct, yes):
$$ \hat y_i = \hat{w}_1 {ReLU}\bigg( \hat w_{red}f_{1,1} + \hat w_{blue}f_{1,2} + \hat w_{purple}f_{2,1} + \hat w_{grey}f_{2,2} \bigg) + \hat{w}_2 {ReLU}\bigg( \hat w_{red}f_{1,2} + \hat w_{blue}f_{1,3} + \hat w_{purple}f_{2,2} + \hat w_{grey}f_{2,3} \bigg) + \hat{w}_3 {ReLU}\bigg( \hat w_{red}f_{2,1} + \hat w_{blue}f_{2,2} + \hat w_{purple}f_{3,1} + \hat w_{grey}f_{3,2} \bigg) + \hat{w}_4 {ReLU}\bigg( \hat w_{red}f_{2,2} + \hat w_{blue}f_{2,3} + \hat w_{purple}f_{3,2} + \hat w_{grey}f_{3,3} \bigg) $$
However, we could have fully-connected the nine features to the four hidden-layer neuros, and that would have connected each circle on the left to each circle on the right; for instance, the convolutional neural network lacks a connection between feature $f_{1,1}$ and hidden neuron $4$. In other words, we force that weight to be zero. Also, we force certain connections to be equal. In an unconstrained, fully-connected neural network, there would be $36$ distinct colors, rather than just my $4$.
While the idea of a CNN as using filters to mimic how our eyes and brains break down the components of an image gives good intuition, from the standpoint of statistics, that seems extreme. It seems like we could have some kind of penalty that makes the model want to make certain connections be close to (or exactly) equal, such as $f_{1,1}\rightarrow \#1$ and $f_{2,1}\rightarrow \#3$, and certain connections be close to (or exactly) zero, such as $f_{1,1}\rightarrow \#4$.
I propose something like the following (again, omitting the bias terms for the sake of clarity).
$$ \hat y_i = \hat{w}_1 {ReLU}\bigg( \hat w_{1,1,1}f_{1,1} + \hat w_{1,2,1}f_{1,2} + \hat w_{1,3,1}f_{1,3} + \hat w_{2,1,1}f_{2,1} + \hat w_{2,2,1}f_{2,2} + \hat w_{2,3,1}f_{2,3} + \hat w_{3,1,1}f_{3,1} + \hat w_{3,2,1}f_{3,2} + \hat w_{3,3,1}f_{3,3} \bigg) + \\ \hat{w}_2 {ReLU}\bigg( \hat w_{1,1,2}f_{1,1} + \hat w_{1,2,2}f_{1,2} + \hat w_{1,3,2}f_{1,3} + \hat w_{2,1,2}f_{2,1} + \hat w_{2,2,2}f_{2,2} + \hat w_{2,3,2}f_{2,3} + \hat w_{3,1,2}f_{3,1} + \hat w_{3,2,2}f_{3,2} + \hat w_{3,3,2}f_{3,3} \bigg) + \\ \hat{w}_3 {ReLU}\bigg( \hat w_{1,1,3}f_{1,1} + \hat w_{1,2,3}f_{1,2} + \hat w_{1,3,3}f_{1,3} + \hat w_{2,1,3}f_{2,1} + \hat w_{2,2,3}f_{2,2} + \hat w_{2,3,3}f_{2,3} + \hat w_{3,1,3}f_{3,1} + \hat w_{3,2,3}f_{3,2} + \hat w_{3,3,3}f_{3,3} \bigg) + \\ \hat{w}_4 {ReLU}\bigg( \hat w_{1,1,4}f_{1,1} + \hat w_{1,2,4}f_{1,2} + \hat w_{1,3,4}f_{1,3} + \hat w_{2,1,4}f_{2,1} + \hat w_{2,2,4}f_{2,2} + \hat w_{2,3,4}f_{2,3} + \hat w_{3,1,4}f_{3,1} + \hat w_{3,2,4}f_{3,2} + \hat w_{3,3,4}f_{3,3} \bigg) $$
Then, like in ridge and LASSO regression, the loss function will include some penalty term(s) for having values like $\hat w_{1,1,2}$ not equal to zero and values like $\hat w_{1,1,1}$ and $\hat w_{1,2,2}$ not equal to each other.
Has any literature explored an idea like this? My impression is that it is a mild CNN that is more akin to ridge regression than LASSO in that it just penalizes terms (or difference in terms) for being nonzero in an attempt to make them small, rather than squashing them down to exactly zero.
Perhaps, instead of winding up with four colors like in my drawings, we could wind up with four color families of the Reds, the Blues, the Purples, and the Greys (and the Whites for the nearly-dropped connections), not all of which are exactly the same shade of their family color but certainly have hues more similar to their own family than to other families.
(Or maybe the data will say that there should be a problem-child in some family, multiple in multiple families, or absolutely no family identity in terms of hue. The point is that the data would guide such a decision like it does in ridge regression, rather than forcing all of the Reds, for instance, to be the same color of red.)
