4
$\begingroup$

I am currently studying the paper of network implementation RCNN.

The core module inside RCNN is the Recurrent Convolutional Layer (RCL), whose state evolves over discrete time steps.

The network is similar to ResNet.

Each RCL block is defined as such...

def RCL(feed_forward_input,num_of_filter, filtersize, alpha,pool):
    conv = Conv2D(filters=num_of_filter, kernel_size=filtersize, padding='same')
    recurrent_input = conv(feed_forward_input) #Yes I could have used a less confusing name... 
    merged = add([feed_forward_input,recurrent_input])
    conv_relu = Activation(create_relu_advanced(alpha_val=alpha))(merged)
    conv_relu_batchnorm = BatchNormalization()(conv_relu)
    if pool:
        conv_relu_batchnorm_pool = MaxPooling2D()(conv_relu_batchnorm)
        return conv_relu_batchnorm_pool
    else:

        return conv_relu_batchnorm

What I don't get is why batch normalization is done at the end and not at the beginning? Would it make sense to normalize the input? It makes sense to normalize the input, but does it make sense to normalize the output of each layer?

What I guess confuses me, is it's usually used at the beginning, so doing it at the end seems a bit... unusual? - as described here and here and also here.

$σ(x) = f_n(g(x))$ is a composition of two nonlinear functions. The inner one $g(x)$ can be either a conventional sigmoid function $g(x) = > 1/(1 + e−x)$ or a rectified linear unit (ReLU) [21] $g(x) = max{x, > 0}$. A model with ReLU usually converges faster and tends to achieve better performance compared to using the sig- moid function. However, the faster convergence brings the problem of “exploding gradient”, which calls for smaller learning rate and necessary normalization. The outer function $f_n(·)$ denotes an appropriate normalization function. The batch-normalization method [14] is adopted here

Its done after Relu... weird? .

$\endgroup$
7
$\begingroup$

According to Ioffe and Szegedy (2015), batch normalization is employed to stabilize the inputs to nonlinear activation functions.

"Batch Normalization seeks a stable distribution of activation values throughout training, and normalizes the inputs of a nonlinearity since that is where matching the moments is more likely to stabilize the distribution"

So normally, it is inserted after dense layers and before the nonlinearity. Below is a part of lecture notes for CS231n.

enter image description here

$\endgroup$
  • $\begingroup$ Sooo? why are they doing it? $\endgroup$ – Bob Burt Sep 8 '17 at 10:26
  • $\begingroup$ The paper states, that it should be done after the relu?.. I've been using the network for a while now.. 2-3 months, and it seems to perform pretty well.. Not tested with their benchmark though. $\endgroup$ – Bob Burt Sep 8 '17 at 10:31
  • 2
    $\begingroup$ You could refer to this page. They show some interesting experiment results and interpretations: github.com/ducha-aiki/caffenet-benchmark/blob/master/… $\endgroup$ – Buomsoo Kim Sep 8 '17 at 10:37
  • 1
    $\begingroup$ They say that BN after relu is rather effective, interestingly $\endgroup$ – Buomsoo Kim Sep 8 '17 at 10:38
  • 2
    $\begingroup$ What about after MaxPool? $\endgroup$ – insanely_sin Feb 11 '19 at 4:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.