# Weight normalization technique used in Image Style Transfer

I am trying to implement the paper Image Style Transfer Using Convolutional Neural Networks. In section 2 - Deep image representations, the authors mention the following weight normalization technique:

We normalized the network by scaling the weights such that the mean activation of each convolutional ﬁlter over images and positions is equal to one. Such re-scaling can be done for the VGG network without changing its output, because it contains only rectifying linear activation functions and no normalization or pooling over feature maps.

From a related question asked earlier I found that the authors are using the activation values from images of ILSVRC validation set to normalize the weights.

I wanted to know the mathematical formulation of performing such a normalization as I was unable to come up with one myself.

According to my understanding of the problem I have an activation map (X) and corresponding to it I have K activation maps of the previous layer(L) and a weight matrix (W) of dimensions 3x3xK, such that when layer L is convolved with W it produces X. Now, once I captured the activation values for all the neurons in layer L for all images in the validation set, the objective is to make the mean of all neurons in X across all images in the validation set equal to 1 by somehow adjusting W.

I could not figure out what should I do to W to make that happen.

Also, I wanted to know if this is to be performed in a cascaded(sequential) way by normalizing weights of the initial layer first and then using the new feature maps to normalize the weights of the layers ahead or independently for each activation map by taking the values of previous layer as the original pre-trained weights for every activation map?

Short answer: Take the activation map corresponding to a particular weight matrix, take the mean of all the activations, and then average this mean over all images. Then divide the weight matrix and the bias by this average. And yes I'd assume it has to be done sequentially.

Long answer: (Using the notation used in the paper you cited)

The convolution operator for the $$i^{th}$$ feature map performs an inner product with image patches $$x_j$$:

$$max\{0,\ w_i^{l} \bullet x_{j} + b_j^{l}\} = F_{ij}^l$$

They take the mean of activations over all images $$\chi$$ and all spatial locations $$j$$ (let's call that $$s_i$$)

$$s_i^{l} \equiv \mathbf{E}_{\chi, j}[max\{0,\ w_i^{l} \bullet x_{j} + b_j^{l}\}] = \frac{1}{KM_l} \sum_{\chi} \sum_{j=1}^{M_l} F_{ij}^l$$

Here $$K$$ is the number of images in the dataset.

Now you just scale $$w_i^{l}$$ and $$b_j^{l}$$ by $$\frac{1}{s_i^{l}}$$, giving you:

$$\mathbf{E}_{\chi, j}[ max\{0,\ \frac{w_i^{l}}{s_i^{l}} \bullet x_{j} + \frac{b_j^{l}}{s_i^{l}}\}] = 1$$

This also ensures that activations that were zero earlier, after passing through the RELU nonlinearity, remain so, i.e.

$$w_i^{l} \bullet x_{j} + b_j^{l}< 0 \iff \frac{w_i^{l}}{s_i^{l}} \bullet x_{j} + \frac{b_j^{l}}{s_i^{l}}< 0$$