Importance of local response normalization in CNN

Question

I've found that Imagenet and other large CNN makes use of local response normalization layers. However, I cannot find that much information about them. How important are they and when should they be used?

From http://caffe.berkeleyvision.org/tutorial/layers.html#data-layers:

"The local response normalization layer performs a kind of “lateral inhibition” by normalizing over local input regions. In ACROSS_CHANNELS mode, the local regions extend across nearby channels, but have no spatial extent (i.e., they have shape local_size x 1 x 1). In WITHIN_CHANNEL mode, the local regions extend spatially, but are in separate channels (i.e., they have shape 1 x local_size x local_size). Each input value is divided by (1+(α/n)∑ix2i)β, where n is the size of each local region, and the sum is taken over the region centered at that value (zero padding is added where necessary)."

Edit:

It seems that these kinds of layers have a minimal impact and are not used any more. Basically, their role have been outplayed by other regularization techniques (such as dropout and batch normalization), better initializations and training methods. See my answer below for more details.

Answer 1

It seems that these kinds of layers have a minimal impact and are not used any more. Basically, their role have been outplayed by other regularization techniques (such as dropout and batch normalization), better initializations and training methods. This is what is written in the lecture notes for the Stanford Course CS321n on ConvNets:

Normalization Layer

Many types of normalization layers have been proposed for use in ConvNet architectures, sometimes with the intentions of implementing inhibition schemes observed in the biological brain. However, these layers have recently fallen out of favor because in practice their contribution has been shown to be minimal, if any. For various types of normalizations, see the discussion in Alex Krizhevsky's cuda-convnet library API.

Answer 2

Indeed, there seems no good explanation in a single place. The best is to read the articles from where it comes:

The original AlexNet article explains a bit in Section 3.3:

• Krizhevsky, Sutskever, and Hinton, ImageNet Classification with Deep Convolutional Neural Networks, NIPS 2012. pdf

The exact way of doing this was proposed in (but not much extra info here):

• Kevin Jarrett, Koray Kavukcuoglu, Marc’Aurelio Ranzato and Yann LeCun, What is the best Multi-Stage Architecture for Object Recognition?, ICCV 2009. pdf

It was inspired by computational neuroscience:

• S. Lyu and E. Simoncelli. Nonlinear image representation using divisive normalization. CVPR 2008. pdf. This paper goes deeper into the math, and is in accordance with the answer of seanv507.
• [24] N. Pinto, D. D. Cox, and J. J. DiCarlo. Why is real-world vi- sual object recognition hard? PLoS Computational Biology, 2008.

Answer 3

Here is my suggested answer, though I don't claim to be knowledgeable. When performing gradient descent on a linear model, the error surface is quadratic, with the curvature determined by $XX_T$, where $X$ is your input. Now the ideal error surface for or gradient descent has the same curvature in all directions (otherwise the step size is too small in some directions and too big in others). Normalising your inputs by rescaling the inputs to mean zero, variance 1 helps and is fast:now the directions along each dimension all have the same curvature, which in turn bounds the curvature in other directions.

The optimal solution would be to sphere/whiten the inputs to each neuron, however this is computationally too expensive. LCN can be justified as an approximate whitening based on the assumption of a high degree of correlation between neighbouring pixels (or channels) So I would claim the benefit is that the error surface is more benign for SGD... A single Learning rate works well across the input dimensions (of each neuron)

Answer 4

With this answer I would like to summarize contributions of other authors and provide a single place explanation of the LRN (or contrastive normalization) technique for those, who just want to get aware of what it is and how it works.

Motivation: 'This sort of response normalization (LRN) implements a form of lateral inhibition inspired by the type found in real neurons, creating competition for big activities among neuron outputs computed using different kernels.' AlexNet 3.3

In other words LRN allows to diminish responses that are uniformly large for the neighborhood and make large activation more pronounced within a neighborhood i.e. create higher contrast in activation map. prateekvjoshi.com states that it is particulary useful with unbounded activation functions as RELU.

Original Formula: For every particular position (x, y) and kernel i that corresponds to a single 'pixel' output we apply a 'filter', that incorporates information about outputs of other n kernels applied to the same position. This regularization is applied before activation function. This regularization, indeed, relies on the order of kernels which is, to my best knowledge, just an unfortunate coincidence.

In practice (see Caffe) 2 approaches can be used:

1. WITHIN_CHANNEL. Normalize over local neighborhood of a single channel (corresponding to a single convolutional filter). In other words, divide response of a single channel of a single pixel according to output values of the same neuron for pixels nearby.
2. ACROSS_CHANNELS. For a single pixel normalize values of every channel according to values of all channels for the same pixel

Actual usage LRN was used more often during the days of early convets like LeNet-5. Current implementation of GoogLeNet (Inception) in Caffe often uses LRN in connection with pooling techniques, but it seems to be done for the sake of just having it. Neither original Inception/GoogLeNet (here) nor any of the following versions mention LRN in any way. Also, TensorFlow implementation of Inception (provided and updated by the team of original authors) networks does not use LRN despite it being available.

Conclusion Applying LRN along with pooling layer would not hurt the performance of the network as long as hyper-parameter values are reasonable. Despite that, I am not aware of any recent justification for applying LRN/contrast normalization in a neural-network.

Answer 5

Local Response Normalization(LRN) type of layer turns out to be useful when using neurons with unbounded activations (e.g. rectified linear neurons), because it permits the detection of high-frequency features with a big neuron response, while damping responses that are uniformly large in a local neighborhood. It is a type of regularizer that encourages "competition" for big activities among nearby groups of neurons.

src-https://code.google.com/p/cuda-convnet/wiki/LayerParams#Local_response_normalization_layer_(same_map)

Answer 6

Local response normalization (LRN) is done pixel-wise for each channel $i$:

$$x_i = \frac{x_i}{ (k + ( \alpha \sum_j x_j^2 ))^\beta }$$

where $k, \alpha, \beta \in \mathbb{R}$ are constants. Note that you get L2 normalization if you set $\kappa = 0$, $\alpha=1$, $\beta=\frac{1}{2}$.

However, there is a much newer technique called "batch normalization" (see paper) which works pretty similar and suggests not to use LRN anymore. Batch normalization also works pixel-wise:

$$y = \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} \gamma + \beta$$

where $\mu$ is the mean, $\sigma^2$ is the variance, $\varepsilon > 0$ is a small constant, $\gamma, \beta \in \mathbb{R}$ are learnable parameters which allow the net to remove the normalization.

So the answer is: Local Response Normalization is not important any more, because we have something which works better and replaced LRN: Batch Normalization.

See also

• Lasagne documentation

Answer 7

AlexNet also uses a competitive normalization step immediately after the ReLU step of layers C1 and C3, called local response normalization (LRN): the most strongly activated neurons inhibit other neurons located at the same position in neighboring feature maps (such competitive activation has been observed in biological neurons). This encourages different feature maps to specialize, pushing them apart and forcing them to explore a wider range of features, ultimately improving generalization. Equation 14-2 shows how to apply LRN. Equation 14-2. Local response normalization (LRN) In this equation:

• $b_i$ is the normalized output of the neuron located in feature map $i$, at some row u and column v (note that in this equation we consider only neurons located at this row and column, so u and v are not shown).

• $a_i$ is the activation of that neuron after the ReLU step, but before normalization.

• k, α, β, and r are hyperparameters. k is called the bias, and r is called the depth radius.

• $f_n$ is the number of feature maps.

For example, if r = 2 and a neuron has a strong activation, it will inhibit the activation of the neurons located in the feature maps immediately above and below its own.

In AlexNet, the hyperparameters are set as follows: r = 5, α = 0.0001, β = 0.75, and k = 2. This step can be implemented using the tf.nn.local_response_normalization() function (which you can wrap in a Lambda layer if you want to use it in a Keras model).

A variant of AlexNet called ZF Net12 was developed by Matthew Zeiler and Rob Fergus and won the 2013 ILSVRC challenge. It is essentially AlexNet with a few tweaked hyperparameters (number of feature maps, kernel size, stride, etc.). GoogLeNet[![enter image description here][2]][2]

The answer is from this book Hands-On Machine Learning with Scikit-Learn, Keras, and Tensorflow: Concepts, Tools, and Techniques to Build Intelligent Systems