Usually, when convolving images, the elements in the filter sum to one. Is this criterion enforced in convolutional neural networks? If yes, how?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
No, in convolutional neural networks, there's no criteria that enforces the learned filters to sum to $1$. In fact, there is nothing in the definition of a convolution $f * g$, continuous or discrete, that says the filter $g$ needs to sum to one. You may be thinking of a discrete gaussian kernel for blur operations that often (but not necessarily) sum to one.
Convolving two continuous 2-D functions $f$ and $g$ is defined as [1]:
$$ \int_{\tau_x=-\infty}^\infty \int_{\tau_y=-\infty}^{\infty} f(\tau_x, \tau_y)\, g(x - \tau_x, y - \tau_y)\, d\tau_y\, d\tau_x $$
In machine vision and image processing, we're dealing with discrete images and thus discrete convolutions. Not only are we dealing with discrete images, but the filters that we use are finite filters; for computational efficiency we won't convolve an infinitely large discrete filter over the original image $f$.
A discrete 2-D convolution of a discrete function $f$ and a finite filter $g$ of size $(w, h)$ is defined as [2][3]:
$$ (f * g) [x, y] = \sum_{x_f=0}^{w - 1} \sum_{y_f=0}^{h- 1} f[x - x_f, y - y_f]\,g[x_f,\, y_f] $$
These types of discrete 2-D convolutions are central to the feed forward pass of a convolutional neural network. They're typically implemented with either FFTs and multiplication or matrix multiplication with Toeplitz matricies.[4][5]
As you can see, there's no constraint or rule in the definitions that says that the double improper integral of $g$ from $(-\infty, -\infty)$ to $(\infty, \infty)$ (in the continuous case) or the sum from $(0,0)$ to $(w - 1, h - 1)$ (in the discrete case) needs to equal $1$.
I want to add that there are a variety of normalization steps that can happen before you input images into a convolutional neural network, such as PCA whitening or homomorphic filtering. You can even normalize your network output so that the feature vector sums to one. Some papers ([6]) do this. However, the point is that these normalization steps are not inherent to the convolution operation or CNNs.
- http://en.wikipedia.org/wiki/Convolution#Definition
- http://en.wikipedia.org/wiki/Convolution#Discrete_convolution
- http://www.cim.mcgill.ca/~langer/558/10-edgecorner.pdf
- http://en.wikipedia.org/wiki/Convolution_theorem
- http://en.wikipedia.org/wiki/Toeplitz_matrix#Discrete_convolution
- http://www.cv-foundation.org/openaccess/content_cvpr_2014/papers/Taigman_DeepFace_Closing_the_2014_CVPR_paper.pdf
$\begingroup$
$\endgroup$
As sabalaba points out in his answer, this is not enforced in convolutional neural networks.
Nevertheless, in some applications you may want to enforce unit norm. When training with standard SGD backpropagation, the simplest way to obtain unit-energy filters is to perform a normalization operation after each SGD update.
