To understand DeconvNet, let's start with Saliency map (or Vanilla gradient), the goal is to back-propagate the gradient in order to get an idea of the aspects of the input image that caused a neural network to make a specific prediction.

With an input $x$ a class of interest $c$, and a model $f$, then Saliency map is simply the derivative of $f^c$ with respect to the image $x$

$$ \frac { \partial{f^c} } { \partial{x} } $$

In this way the gradient is propagated backwards until it the network input. Now the backpropagation rule from a layer $l$ to the layer before $l_{-1}$ :

$$ \frac { \partial{f^c} } { \partial{x_{l-1}} } = \frac { \partial{x_{l}} } { \partial{x_{l-1}} } \frac { \partial{f^c} } { \partial{x_{l}} } $$ Going backward performing all the operations of the network (Unpooling, Filtering...), and for non-linearities, only pass gradients to regions of positive activations $ R_{l} = 1_{z_l > 0}\ R_{l+1} $.

So far all we're doing is backpropagating the gradient by reversing the operations. But the way DeconvNet handle the non-linearities is different as they propose to only propagate positive gradient, $ R_{l} = 1_{R_{l+1} > 0}\ R_{l+1} $ or $ R_{l} = ReLU(R_{l+1}) $ .

Here is a simple example, we start by the forward pass:

$$ x_l \begin{pmatrix} 1 & -2 \\ -4 & 5 \end{pmatrix} \rightarrow z_l \begin{pmatrix} 1 & 0 \\ 0 & 5 \end{pmatrix} \\ $$ Now, with $R_l$ our intermediate backpropagation result
$$ R_l \begin{pmatrix} -2 & -2 \\ 4 & 8 \end{pmatrix} $$

We have two possibilities to obtain $R_{l-1}$, using basic Saliency method (only take gradients from positive region): $$ R_l \begin{pmatrix} -2 & -2 \\ 4 & 8 \end{pmatrix} \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix} \rightarrow \begin{pmatrix} -2 & 0 \\ 0 & 8 \end{pmatrix} $$ Using DeconvNet method (only take positive gradients) : $$ R_l \begin{pmatrix} -2 & -2 \\ 4 & 8 \end{pmatrix} \begin{pmatrix} 0 & 0 \\ 1 & 1 \end{pmatrix} \rightarrow \begin{pmatrix} 0 & 0 \\ 4 & 8 \end{pmatrix} $$

To understand DeconvNet, let's start with Saliency map (or Vanilla gradient), the goal is to back-propagate the gradient in order to get an idea of the aspects of the input image that caused a neural network to make a specific prediction.

With an input $x$ a class of interest $c$, and a model $f$, then Saliency map is simply the derivative of $f^c$ with respect to the image $x$

$$ \frac { \partial{f^c} } { \partial{x} } $$

In this way the gradient is propagated backwards until it the network input. Now the backpropagation rule from a layer $l$ to the layer before $l_{-1}$ :

$$ \frac { \partial{f^c} } { \partial{x_{l-1}} } = \frac { \partial{x_{l}} } { \partial{x_{l-1}} } \frac { \partial{f^c} } { \partial{x_{l}} } $$ Going backward performing all the operations of the network (Unpooling, Filtering...), and for non-linearities, only pass gradients to regions of positive activations $ R_{l} = 1_{z_l > 0}\ R_{l+1} $.

So far all we're doing is backpropagating the gradient by reversing the operations. But the way DeconvNet handle the non-linearities is different as they propose to only propagate positive gradient, $ R_{l} = 1_{R_{l+1} > 0}\ R_{l+1} $ or $ R_{l} = ReLU(R_{l+1}) $ .

Here is a simple example, we start by the forward pass:

$$ x_l \begin{pmatrix} 1 & -2 \\ -4 & 5 \end{pmatrix} \rightarrow z_l \begin{pmatrix} 1 & 0 \\ 0 & 5 \end{pmatrix} \\ $$ Now, with $R_l$ our intermediate backpropagation result
$$ R_l \begin{pmatrix} -2 & -2 \\ 4 & 8 \end{pmatrix} $$

We have two possibilities to obtain $R_{l-1}$, using basic Saliency method (only take gradients from positive region): $$ R_l \odot 1_{z_l > 0} \\ \begin{pmatrix} -2 & -2 \\ 4 & 8 \end{pmatrix} \odot \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix} \rightarrow \begin{pmatrix} -2 & 0 \\ 0 & 8 \end{pmatrix} $$ Using DeconvNet method (only take positive gradients) : $$ ReLU(R_l) \\ \begin{pmatrix} -2 & -2 \\ 4 & 8 \end{pmatrix} \odot \begin{pmatrix} 0 & 0 \\ 1 & 1 \end{pmatrix} \rightarrow \begin{pmatrix} 0 & 0 \\ 4 & 8 \end{pmatrix} $$

with $\odot$ the Hadamard (or element wise) product