# Why do DeconvNet use ReLU in the backward pass?

Why does DeconvNet (Zeiler, 2014) use ReLU in the backward pass (after unpooling)? Are not the feature maps values already positive due to the ReLU in the forward pass? So, why do the authors apply the ReLU again coming back to the input?

update: I better explain my problem:

given an input image $$x$$ and ConvLayer $$CL$$ composed of:

1. a convolution
2. an activation function ReLU
3. a pool operation

$$f$$ is the output of ConvLayer given an input $$x$$, i.e. $$f=CL(x)$$.

So, the Deconv target is to "reverse" the output $$f$$ (the feature map) to restore an approximate version of $$x$$. To this aim, the authors define a function $$CL^{-1}$$ composed of 3 subfunctions:

a. unpool

b. activation function ReLU (useless in my opinion, because $$f$$ is already positive due to the application of the 2. step in $$CL(f)$$)

c. transposed convolution.

In other words $$x\simeq CL^{-1}(f)$$ where $$CL^{-1} (f) = transpconv(relu(unpool(f)))$$. But, if $$f$$ is the output computed as $$f=CL(x)$$, it is already positive, so the b. step is useless.

This is what I understood from the paper. Where I wrong?

## 1 Answer

To understand DeconvNet, let's start with Saliency map (or Vanilla gradient), the goal is to back-propagate the gradient in order to get an idea of the aspects of the input image that caused a neural network to make a specific prediction.

With an input $$x$$ a class of interest $$c$$, and a model $$f$$, then Saliency map is simply the derivative of $$f^c$$ with respect to the image $$x$$

$$\frac { \partial{f^c} } { \partial{x} }$$

In this way the gradient is propagated backwards until it the network input. Now the backpropagation rule from a layer $$l$$ to the layer before $$l_{-1}$$ :

$$\frac { \partial{f^c} } { \partial{x_{l-1}} } = \frac { \partial{x_{l}} } { \partial{x_{l-1}} } \frac { \partial{f^c} } { \partial{x_{l}} }$$ Going backward performing all the operations of the network (Unpooling, Filtering...), and for non-linearities, only pass gradients to regions of positive activations $$R_{l} = 1_{z_l > 0}\ R_{l+1}$$.

So far all we're doing is backpropagating the gradient by reversing the operations. But the way DeconvNet handle the non-linearities is different as they propose to only propagate positive gradient, $$R_{l} = 1_{R_{l+1} > 0}\ R_{l+1}$$ or $$R_{l} = ReLU(R_{l+1})$$ .

Here is a simple example, we start by the forward pass:

$$x_l \begin{pmatrix} 1 & -2 \\ -4 & 5 \end{pmatrix} \rightarrow z_l \begin{pmatrix} 1 & 0 \\ 0 & 5 \end{pmatrix} \\$$ Now, with $$R_l$$ our intermediate backpropagation result
$$R_l \begin{pmatrix} -2 & -2 \\ 4 & 8 \end{pmatrix}$$

We have two possibilities to obtain $$R_{l-1}$$, using basic Saliency method (only take gradients from positive region): $$R_l \odot 1_{z_l > 0} \\ \begin{pmatrix} -2 & -2 \\ 4 & 8 \end{pmatrix} \odot \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix} \rightarrow \begin{pmatrix} -2 & 0 \\ 0 & 8 \end{pmatrix}$$ Using DeconvNet method (only take positive gradients) : $$ReLU(R_l) \\ \begin{pmatrix} -2 & -2 \\ 4 & 8 \end{pmatrix} \odot \begin{pmatrix} 0 & 0 \\ 1 & 1 \end{pmatrix} \rightarrow \begin{pmatrix} 0 & 0 \\ 4 & 8 \end{pmatrix}$$

with $$\odot$$ the Hadamard (or element wise) product

• tnx for the anwer! So, DecovNet acts differently from what I understood? I update the answer with what I understood from the paper. Is the pipeline I described different from what you said? From your explanation, it seems that the Deconv input is not $f$, but something made during the backpropagation from the forward level (the gradient error?). – volperossa Mar 30 '20 at 17:51
• Exactly, the 'Deconv input' is the error signal, not the output of the model – Thomas FEL Mar 30 '20 at 19:22
• so the error signal is "unpooled" and not the pooling-layer output? It seems to me without sense... – volperossa Mar 30 '20 at 20:13
• from the paper: "To start, an input image is presented to the convnet and features computed throughout the layers. To examine a given convnet activation, we set all other activations in the layer to zero and pass the feature maps as input to the attached deconvnet layer" <---- the feature map is passed as input.... – volperossa Mar 30 '20 at 20:55
• Ok sorry I wasn't very clear, I mean, of course to compute the error signal you need the output of the model, but the error signal is not the output of the model. Take a look at the equation 2, and apply it for layer $l$ , its $\frac{\partial{y^c}}{\partial{x_l}}$, the gradient tells you how the last layer should be to maximize your prediction ($y^c$ to 1), that's the signal (you can also take the negative of a loss function like cross-entropy), you then go back up by applying the rules described above until you reach the input – Thomas FEL Apr 1 '20 at 9:45