Why do DeconvNet use ReLU in the backward pass?

Question

Why does DeconvNet (Zeiler, 2014) use ReLU in the backward pass (after unpooling)? Are not the feature maps values already positive due to the ReLU in the forward pass? So, why do the authors apply the ReLU again coming back to the input?

ref: https://arxiv.org/abs/1311.2901

update: I better explain my problem:

given an input image $x$ and ConvLayer $CL$ composed of:

1. a convolution
2. an activation function ReLU
3. a pool operation

$f$ is the output of ConvLayer given an input $x$, i.e. $f=CL(x)$.

So, the Deconv target is to "reverse" the output $f$ (the feature map) to restore an approximate version of $x$. To this aim, the authors define a function $CL^{-1}$ composed of 3 subfunctions:

a. unpool

b. activation function ReLU (useless in my opinion, because $f$ is already positive due to the application of the 2. step in $CL(f)$)

c. transposed convolution.

In other words $x\simeq CL^{-1}(f)$ where $CL^{-1} (f) = transpconv(relu(unpool(f)))$. But, if $f$ is the output computed as $f=CL(x)$, it is already positive, so the b. step is useless.

This is what I understood from the paper. Where I wrong?

Answer 1

To understand DeconvNet, let's start with Saliency map (or Vanilla gradient), the goal is to back-propagate the gradient in order to get an idea of the aspects of the input image that caused a neural network to make a specific prediction.

With an input $x$ a class of interest $c$, and a model $f$, then Saliency map is simply the derivative of $f^c$ with respect to the image $x$

$$ \frac { \partial{f^c} } { \partial{x} } $$

In this way the gradient is propagated backwards until it the network input. Now the backpropagation rule from a layer $l$ to the layer before $l_{-1}$ :

$$ \frac { \partial{f^c} } { \partial{x_{l-1}} } = \frac { \partial{x_{l}} } { \partial{x_{l-1}} } \frac { \partial{f^c} } { \partial{x_{l}} } $$ Going backward performing all the operations of the network (Unpooling, Filtering...), and for non-linearities, only pass gradients to regions of positive activations $ R_{l} = 1_{z_l > 0}\ R_{l+1} $.

So far all we're doing is backpropagating the gradient by reversing the operations. But the way DeconvNet handle the non-linearities is different as they propose to only propagate positive gradient, $ R_{l} = 1_{R_{l+1} > 0}\ R_{l+1} $ or $ R_{l} = ReLU(R_{l+1}) $ .

Here is a simple example, we start by the forward pass:

$$ x_l \begin{pmatrix} 1 & -2 \\ -4 & 5 \end{pmatrix} \rightarrow z_l \begin{pmatrix} 1 & 0 \\ 0 & 5 \end{pmatrix} \\ $$ Now, with $R_l$ our intermediate backpropagation result
$$ R_l \begin{pmatrix} -2 & -2 \\ 4 & 8 \end{pmatrix} $$

We have two possibilities to obtain $R_{l-1}$, using basic Saliency method (only take gradients from positive region): $$ R_l \odot 1_{z_l > 0} \\ \begin{pmatrix} -2 & -2 \\ 4 & 8 \end{pmatrix} \odot \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix} \rightarrow \begin{pmatrix} -2 & 0 \\ 0 & 8 \end{pmatrix} $$ Using DeconvNet method (only take positive gradients) : $$ ReLU(R_l) \\ \begin{pmatrix} -2 & -2 \\ 4 & 8 \end{pmatrix} \odot \begin{pmatrix} 0 & 0 \\ 1 & 1 \end{pmatrix} \rightarrow \begin{pmatrix} 0 & 0 \\ 4 & 8 \end{pmatrix} $$

with $\odot$ the Hadamard (or element wise) product