3
$\begingroup$

How to up-sample gradients, during back-propagation, across an average-pooling layer?

For this purpose, let

$$ A^{[l]} = \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \\ a_{31} & a_{32} & a_{33} \\ \end{bmatrix}; \quad P^{[l]} = \begin{bmatrix} p_{11} & p_{12} \\ p_{21} & p_{22} \\ \end{bmatrix} = \begin{bmatrix} \frac{a_{11} + a_{12} + a_{21} + a_{22}}{4} & \frac{a_{12} + a_{13} + a_{22} + a_{23}}{4} \\ \frac{a_{21} + a_{22} + a_{31} + a_{32}}{4} & \frac{a_{22} + a_{23} + a_{32} + a_{33}}{4} \\ \end{bmatrix} \\ \frac{\mathrm{d}J}{\mathrm{d}P^{[l]}} = \begin{bmatrix} \frac{\mathrm{d}J}{\mathrm{d}p_{11}} & \frac{\mathrm{d}J}{\mathrm{d}p_{12}} \\ \frac{\mathrm{d}J}{\mathrm{d}p_{21}} & \frac{\mathrm{d}J}{\mathrm{d}p_{22}} \\ \end{bmatrix} $$

where, $A^{[l]}$ is the activation of layer-$l$ , and $P^{[l]}$ is the matrix obtained after average-pooling $A^{[l]}$ using a $2 \times 2$ pooling window, and $J$ is the cost function.

Given this, how to compute $\frac{\mathrm{d}J}{\mathrm{d}A^{[l]}}$?

  • (Source) says, "the error is multiplied by $\frac{1}{2×2}$ and assigned to the whole pooling block (all units get this same value)." $\rightarrow$ then what would be the gradient $\frac{\mathrm{d}J}{\mathrm{d}a_{22}}$, which is a member of all the pooling blocks.
  • (Source) the question mentions the up-sampling strategy, but doesn't mention what $\beta$ is.
  • (Source) says, "if we have mean pooling then upsample simply uniformly distributes the error for a single pooling unit among the units which feed into it in the previous layer." $\rightarrow$ which is still vague.
  • (Source) Zhang, Zhifei. "Derivation of backpropagation in convolutional neural network (cnn)." University of Tennessee, Knoxville, TN (2016). $\rightarrow$ pg.4, eq.32 results in the following up-sampled gradients $$ \frac{\mathrm{d}J}{\mathrm{d}A^{[l]}} = \begin{bmatrix} \frac{1}{4}\frac{\mathrm{d}J}{\mathrm{d}p_{11}} & \frac{1}{4}\frac{\mathrm{d}J}{\mathrm{d}p_{11}} & \frac{1}{4}\frac{\mathrm{d}J}{\mathrm{d}p_{12}} \\ \frac{1}{4}\frac{\mathrm{d}J}{\mathrm{d}p_{11}} & \frac{1}{4}\frac{\mathrm{d}J}{\mathrm{d}p_{11}} & \frac{1}{4}\frac{\mathrm{d}J}{\mathrm{d}p_{12}} \\ \frac{1}{4}\frac{\mathrm{d}J}{\mathrm{d}p_{21}} & \frac{1}{4}\frac{\mathrm{d}J}{\mathrm{d}p_{21}} & \frac{1}{4}\frac{\mathrm{d}J}{\mathrm{d}p_{12}} \\ \end{bmatrix} $$ Is this correct? And, if it is, then is there a mathematical basis for why the gradients are up-sampled this way?
$\endgroup$

2 Answers 2

1
$\begingroup$

It is not right. When the pool windows overlap, derivatives must be added. This is not explicitly stated in Sources 1-3. I couldn't see it in Source 4 as well but I'm not too sure as the document is long.

Addition happens simply because of the chain rule: $$\frac{\partial J}{\partial a_{ij}}=\sum_{m,n}\frac{\partial J}{\partial p_{mn}}\frac{\partial p_{mn}}{\partial a_{ij}}$$

For example, for $a_{12}$, we'll have

$$\begin{align}\frac{\partial J}{\partial a_{12}}& =\frac{\partial J}{\partial p_{11}}\frac{\partial p_{11}}{\partial a_{12}} +\frac{\partial J}{\partial p_{12}}\frac{\partial p_{12}}{\partial a_{12}} +\frac{\partial J}{\partial p_{21}}\frac{\partial p_{21}}{\partial a_{12}} +\frac{\partial J}{\partial p_{22}}\frac{\partial p_{22}}{\partial a_{12}}\\ &=\frac{\partial J}{\partial p_{11}}\frac{1}{4} +\frac{\partial J}{\partial p_{12}}\frac{1}{4} +\frac{\partial J}{\partial p_{21}}0 +\frac{\partial J}{\partial p_{22}}0\\ &=\frac{\partial J}{\partial p_{11}}\frac{1}{4} +\frac{\partial J}{\partial p_{12}}\frac{1}{4} \end{align}$$

It doesn't matter if this is pooling, upsampling/downsampling etc. We have a bunch of numbers $a_{ij}$, we transform and calculate a bunch of another numbers $p_{mn}$ and propagate until the loss is calculated. This should be treated like a usual transformation in order to comply with the mathematics.

$\endgroup$
3
  • $\begingroup$ how to obtain this summation (i.e., $\frac{\partial J}{\partial a_{ij}}=\sum_{m,n}\frac{\partial J}{\partial p_{mn}}\frac{\partial p_{mn}}{\partial a_{ij}}$) using matrix-calculus? $\endgroup$
    – x.projekt
    Feb 20, 2022 at 7:08
  • 1
    $\begingroup$ I don't know which way would be best, but it needs reshaping since the second multiplicand is a 4d tensor with indices m,n,i,j. If you flatten dJ/dp (row-major) and construct another matrix with shape 4x9 where each entry holds the derivative of dP/dA (both flattened), then you can form a matrix mult, (dJ/dP) x (dP/dA). $\endgroup$
    – gunes
    Feb 20, 2022 at 9:01
  • $\begingroup$ @gunes- I thought of the exact same method, and posted it as an answer. $\endgroup$
    – x.projekt
    Feb 20, 2022 at 9:33
2
$\begingroup$

This is based on gunes's answer above.

We can vectorize all the matrices as follows:

$$ \mathrm{vec}(A^{[l]}) = \begin{bmatrix} a_{11} \\ a_{21} \\ a_{31} \\ a_{12} \\ a_{22} \\ a_{32} \\ a_{13} \\ a_{23} \\ a_{33} \\ \end{bmatrix}; \qquad \mathrm{vec}(P^{[l]}) = \begin{bmatrix} p_{11} \\ p_{21} \\ p_{12} \\ p_{22} \\ \end{bmatrix}\\ \qquad \frac{\mathrm{d}J}{\mathrm{d}\{\mathrm{vec}(P^{[l]})\}} = \begin{bmatrix} \frac{\mathrm{d}J}{\mathrm{d}p_{11}} & \frac{\mathrm{d}J}{\mathrm{d}p_{21}} & \frac{\mathrm{d}J}{\mathrm{d}p_{12}} & \frac{\mathrm{d}J}{\mathrm{d}p_{22}} \end{bmatrix} \qquad\text{eq.1} $$

then compute the gradient $\frac{\mathrm{d}J}{\mathrm{d}A^{[l]}}$, as follows:

Note: I'm using the numerator-layout, so, the derivatives are technically Jacobians.

$$ \frac{\mathrm{d}J}{\mathrm{d}\{\mathrm{vec}(A^{[l]})\}} = \frac{\mathrm{d}J}{\mathrm{d}\{\mathrm{vec}(P^{[l]})\}} \frac{\mathrm{d}\{\mathrm{vec}(P^{[l]})\}}{\mathrm{d}\{\mathrm{vec}(A^{[l]})\}} \qquad\text{eq.2}\\ $$

here, $$ \begin{align} \frac{\mathrm{d}\{\mathrm{vec}(P^{[l]})\}}{\mathrm{d}\{\mathrm{vec}(A^{[l]})\}} & = \begin{bmatrix} \frac{\partial}{\partial a_{11}} & \frac{\partial}{\partial a_{21}} & \frac{\partial}{\partial a_{31}} & \frac{\partial}{\partial a_{12}} & \frac{\partial}{\partial a_{22}} & \frac{\partial}{\partial a_{32}} & \frac{\partial}{\partial a_{13}} & \frac{\partial}{\partial a_{23}} & \frac{\partial}{\partial a_{33}} \end{bmatrix} \otimes \begin{bmatrix} p_{11} \\ p_{21} \\ p_{12} \\ p_{22} \\ \end{bmatrix} \\ & = \begin{bmatrix} \frac{\partial p_{11}}{\partial a_{11}} & \frac{\partial p_{11}}{\partial a_{21}} & \frac{\partial p_{11}}{\partial a_{31}} & \frac{\partial p_{11}}{\partial a_{12}} & \frac{\partial p_{11}}{\partial a_{22}} & \frac{\partial p_{11}}{\partial a_{32}} & \frac{\partial p_{11}}{\partial a_{13}} & \frac{\partial p_{11}}{\partial a_{23}} & \frac{\partial p_{11}}{\partial a_{33}} \\ \frac{\partial p_{21}}{\partial a_{11}} & \frac{\partial p_{21}}{\partial a_{21}} & \frac{\partial p_{21}}{\partial a_{31}} & \frac{\partial p_{21}}{\partial a_{12}} & \frac{\partial p_{21}}{\partial a_{22}} & \frac{\partial p_{21}}{\partial a_{32}} & \frac{\partial p_{21}}{\partial a_{13}} & \frac{\partial p_{21}}{\partial a_{23}} & \frac{\partial p_{21}}{\partial a_{33}} \\ \frac{\partial p_{12}}{\partial a_{11}} & \frac{\partial p_{12}}{\partial a_{21}} & \frac{\partial p_{12}}{\partial a_{31}} & \frac{\partial p_{12}}{\partial a_{12}} & \frac{\partial p_{12}}{\partial a_{22}} & \frac{\partial p_{12}}{\partial a_{32}} & \frac{\partial p_{12}}{\partial a_{13}} & \frac{\partial p_{12}}{\partial a_{23}} & \frac{\partial p_{12}}{\partial a_{33}} \\ \frac{\partial p_{22}}{\partial a_{11}} & \frac{\partial p_{22}}{\partial a_{21}} & \frac{\partial p_{22}}{\partial a_{31}} & \frac{\partial p_{22}}{\partial a_{12}} & \frac{\partial p_{22}}{\partial a_{22}} & \frac{\partial p_{22}}{\partial a_{32}} & \frac{\partial p_{22}}{\partial a_{13}} & \frac{\partial p_{22}}{\partial a_{23}} & \frac{\partial p_{22}}{\partial a_{33}} \\ \end{bmatrix} \\ & = \begin{bmatrix} \frac{1}{4} & \frac{1}{4} & 0 & \frac{1}{4} & \frac{1}{4} & 0 & 0 & 0 & 0 \\ 0 & \frac{1}{4} & \frac{1}{4} & 0 & \frac{1}{4} & \frac{1}{4} & 0 & 0 & 0 \\ 0 & 0 & 0 & \frac{1}{4} & \frac{1}{4} & 0 & \frac{1}{4} & \frac{1}{4} & 0 \\ 0 & 0 & 0 & 0 & \frac{1}{4} & \frac{1}{4} & 0 & \frac{1}{4} & \frac{1}{4} \\ \end{bmatrix} \qquad\text{eq.3} \end{align} $$

from eq.1, eq.2, and eq.3, we have

$$ \begin{align} \frac{\mathrm{d}J}{\mathrm{d}\{\mathrm{vec}(A^{[l]})\}} & = \begin{bmatrix} \frac{\partial J}{\partial a_{11}} & \frac{\partial J}{\partial a_{21}} & \frac{\partial J}{\partial a_{31}} & \frac{\partial J}{\partial a_{12}} & \frac{\partial J}{\partial a_{22}} & \frac{\partial J}{\partial a_{32}} & \frac{\partial J}{\partial a_{13}} & \frac{\partial J}{\partial a_{23}} & \frac{\partial J}{\partial a_{33}} \\ \end{bmatrix} \\ & = \begin{bmatrix} \frac{1}{4}\frac{\mathrm{d}J}{\mathrm{d}p_{11}} \\ \frac{1}{4}\biggl(\frac{\mathrm{d}J}{\mathrm{d}p_{11}} + \frac{1}{4}\frac{\mathrm{d}J}{\mathrm{d}p_{21}}\biggr) \\ \frac{1}{4}\frac{\mathrm{d}J}{\mathrm{d}p_{21}} \\ \frac{1}{4}\biggl(\frac{\mathrm{d}J}{\mathrm{d}p_{11}} + \frac{1}{4}\frac{\mathrm{d}J}{\mathrm{d}p_{12}}\biggr) \\ \frac{1}{4}\biggl(\frac{\mathrm{d}J}{\mathrm{d}p_{11}} + \frac{\mathrm{d}J}{\mathrm{d}p_{21}} + \frac{\mathrm{d}J}{\mathrm{d}p_{12}} + \frac{\mathrm{d}J}{\mathrm{d}p_{22}}\biggr) \\ \frac{1}{4}\biggl(\frac{\mathrm{d}J}{\mathrm{d}p_{21}} + \frac{\mathrm{d}J}{\mathrm{d}p_{22}}\biggr) \\ \frac{1}{4}\frac{\mathrm{d}J}{\mathrm{d}p_{12}} \\ \frac{1}{4}\biggl(\frac{\mathrm{d}J}{\mathrm{d}p_{12}} + \frac{\mathrm{d}J}{\mathrm{d}p_{22}}\biggr) \\ \frac{1}{4}\frac{\mathrm{d}J}{\mathrm{d}p_{22}} \\ \end{bmatrix}^\intercal \end{align} $$

And, reshaping the above matrix, we get

$$ \frac{\mathrm{d}J}{\mathrm{d}A^{[l]}} = \begin{bmatrix} \frac{1}{4}\frac{\mathrm{d}J}{\mathrm{d}p_{11}} & \frac{1}{4}\biggl(\frac{\mathrm{d}J}{\mathrm{d}p_{11}} + \frac{1}{4}\frac{\mathrm{d}J}{\mathrm{d}p_{21}}\biggr) & \frac{1}{4}\frac{\mathrm{d}J}{\mathrm{d}p_{21}} \\ \frac{1}{4}\biggl(\frac{\mathrm{d}J}{\mathrm{d}p_{11}} + \frac{1}{4}\frac{\mathrm{d}J}{\mathrm{d}p_{12}}\biggr) & \frac{1}{4}\biggl(\frac{\mathrm{d}J}{\mathrm{d}p_{11}} + \frac{\mathrm{d}J}{\mathrm{d}p_{21}} + \frac{\mathrm{d}J}{\mathrm{d}p_{12}} + \frac{\mathrm{d}J}{\mathrm{d}p_{22}}\biggr) & \frac{1}{4}\biggl(\frac{\mathrm{d}J}{\mathrm{d}p_{21}} + \frac{\mathrm{d}J}{\mathrm{d}p_{22}}\biggr) \\ \frac{1}{4}\frac{\mathrm{d}J}{\mathrm{d}p_{12}} & \frac{1}{4}\biggl(\frac{\mathrm{d}J}{\mathrm{d}p_{12}} + \frac{\mathrm{d}J}{\mathrm{d}p_{22}}\biggr) & \frac{1}{4}\frac{\mathrm{d}J}{\mathrm{d}p_{22}} \\ \end{bmatrix} $$

$\endgroup$
1
  • 1
    $\begingroup$ (+1) thanks for materializing it :) $\endgroup$
    – gunes
    Feb 20, 2022 at 9:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.