1
$\begingroup$

Convolutional neural networks (CNNs) do regularization (of sorts) by forcing some weights to be dropped and others to be zero. Borrowing some drawings from another post of mine...

Apply the filter to the upper left 2x2 array.

Apply the filter to the upper left 2x2 array.

Apply the filter to the upper right 2x2 array.

enter image description here

Apply the filter to the bottom left 2x2 array.

enter image description here

Apply the filter to the bottom right 2x2 array.

enter image description here

Here is the entire layer, with the 3x3 input image mapping to four neurons for the four positions in the image where convolution occurs.

enter image description here

If we see the next layer as the output, then the entire equation would be something like this (I am leaving out the biases for the sake of clarity, but they should be there to be $100\%$ correct, yes):

$$ \hat y_i = \hat{w}_1 {ReLU}\bigg( \hat w_{red}f_{1,1} + \hat w_{blue}f_{1,2} + \hat w_{purple}f_{2,1} + \hat w_{grey}f_{2,2} \bigg) + \hat{w}_2 {ReLU}\bigg( \hat w_{red}f_{1,2} + \hat w_{blue}f_{1,3} + \hat w_{purple}f_{2,2} + \hat w_{grey}f_{2,3} \bigg) + \hat{w}_3 {ReLU}\bigg( \hat w_{red}f_{2,1} + \hat w_{blue}f_{2,2} + \hat w_{purple}f_{3,1} + \hat w_{grey}f_{3,2} \bigg) + \hat{w}_4 {ReLU}\bigg( \hat w_{red}f_{2,2} + \hat w_{blue}f_{2,3} + \hat w_{purple}f_{3,2} + \hat w_{grey}f_{3,3} \bigg) $$

However, we could have fully-connected the nine features to the four hidden-layer neuros, and that would have connected each circle on the left to each circle on the right; for instance, the convolutional neural network lacks a connection between feature $f_{1,1}$ and hidden neuron $4$. In other words, we force that weight to be zero. Also, we force certain connections to be equal. In an unconstrained, fully-connected neural network, there would be $36$ distinct colors, rather than just my $4$.

While the idea of a CNN as using filters to mimic how our eyes and brains break down the components of an image gives good intuition, from the standpoint of statistics, that seems extreme. It seems like we could have some kind of penalty that makes the model want to make certain connections be close to (or exactly) equal, such as $f_{1,1}\rightarrow \#1$ and $f_{2,1}\rightarrow \#3$, and certain connections be close to (or exactly) zero, such as $f_{1,1}\rightarrow \#4$.

I propose something like the following (again, omitting the bias terms for the sake of clarity).

$$ \hat y_i = \hat{w}_1 {ReLU}\bigg( \hat w_{1,1,1}f_{1,1} + \hat w_{1,2,1}f_{1,2} + \hat w_{1,3,1}f_{1,3} + \hat w_{2,1,1}f_{2,1} + \hat w_{2,2,1}f_{2,2} + \hat w_{2,3,1}f_{2,3} + \hat w_{3,1,1}f_{3,1} + \hat w_{3,2,1}f_{3,2} + \hat w_{3,3,1}f_{3,3} \bigg) + \\ \hat{w}_2 {ReLU}\bigg( \hat w_{1,1,2}f_{1,1} + \hat w_{1,2,2}f_{1,2} + \hat w_{1,3,2}f_{1,3} + \hat w_{2,1,2}f_{2,1} + \hat w_{2,2,2}f_{2,2} + \hat w_{2,3,2}f_{2,3} + \hat w_{3,1,2}f_{3,1} + \hat w_{3,2,2}f_{3,2} + \hat w_{3,3,2}f_{3,3} \bigg) + \\ \hat{w}_3 {ReLU}\bigg( \hat w_{1,1,3}f_{1,1} + \hat w_{1,2,3}f_{1,2} + \hat w_{1,3,3}f_{1,3} + \hat w_{2,1,3}f_{2,1} + \hat w_{2,2,3}f_{2,2} + \hat w_{2,3,3}f_{2,3} + \hat w_{3,1,3}f_{3,1} + \hat w_{3,2,3}f_{3,2} + \hat w_{3,3,3}f_{3,3} \bigg) + \\ \hat{w}_4 {ReLU}\bigg( \hat w_{1,1,4}f_{1,1} + \hat w_{1,2,4}f_{1,2} + \hat w_{1,3,4}f_{1,3} + \hat w_{2,1,4}f_{2,1} + \hat w_{2,2,4}f_{2,2} + \hat w_{2,3,4}f_{2,3} + \hat w_{3,1,4}f_{3,1} + \hat w_{3,2,4}f_{3,2} + \hat w_{3,3,4}f_{3,3} \bigg) $$

Then, like in ridge and LASSO regression, the loss function will include some penalty term(s) for having values like $\hat w_{1,1,2}$ not equal to zero and values like $\hat w_{1,1,1}$ and $\hat w_{1,2,2}$ not equal to each other.

Has any literature explored an idea like this? My impression is that it is a mild CNN that is more akin to ridge regression than LASSO in that it just penalizes terms (or difference in terms) for being nonzero in an attempt to make them small, rather than squashing them down to exactly zero.

Perhaps, instead of winding up with four colors like in my drawings, we could wind up with four color families of the Reds, the Blues, the Purples, and the Greys (and the Whites for the nearly-dropped connections), not all of which are exactly the same shade of their family color but certainly have hues more similar to their own family than to other families.

(Or maybe the data will say that there should be a problem-child in some family, multiple in multiple families, or absolutely no family identity in terms of hue. The point is that the data would guide such a decision like it does in ridge regression, rather than forcing all of the Reds, for instance, to be the same color of red.)

$\endgroup$
2
  • 2
    $\begingroup$ vaguely recall a technique called "soft weight sharing" that may be similar? cs.toronto.edu/~fritz/absps/sunspots.pdf $\endgroup$ Commented Jan 10, 2022 at 16:40
  • 1
    $\begingroup$ I awarded the bounty, since someone took the time to answer. However, my question remains. $\endgroup$
    – Dave
    Commented Jan 18, 2022 at 16:41

1 Answer 1

0
+50
$\begingroup$

I'm not an expert but reading your question I have some doubt related the "efficiency" of a similar approach. From what I understand your approach to work need a filter of the same size of the input data. But this can be counterproductive if you consider that maybe only some portion of the input data are relevant for the problem you want to solve. Take for example a dataset of images where you want to find a cat. Then you are interested only in a section of the image where the cat is present not in the entire image itself. A filter of the same size of the input image probably will not have the flexibility to identify a cat in different zone of different images.

$\endgroup$
2
  • $\begingroup$ Thank you for responding, and welcome to Cross Validated! The point of having the penalty would be to control the portions of the picture that make it to the next layer’s feature space. Perhaps I need to be more explicit about the penalty I have in mind in my OP. $\endgroup$
    – Dave
    Commented Jan 11, 2022 at 22:30
  • $\begingroup$ But it always remain the fact that you have to penalize some section of the picture in the end. And maybe with some pictures the section that is penalized is the ones that contain the relevant information $\endgroup$ Commented Jan 12, 2022 at 23:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.