Backpropagation gradient of the average

Question

In the Pytorch Udacity course, the following is said at one point:

To calculate the gradients, you need to run the .backward method on a Variable, z for example. This will calculate the gradient for z with respect to x

Following that, it is said in the course material that

$$ \frac{\partial z}{\partial x} = \frac{\partial}{\partial x}\left[\frac{1}{n}\sum_i^n x_i^2\right] = \frac{x}{2} $$

I would have assumed that one could only take the derivative with respect to an individual $x_i$ since each of them is a separate random variable. However, then the solution would not make sense. I'd be very thankful for some clarity on how to arrive at $x/2$ in this case.

Answer 1

If $x$ is a vector, $z$ is a scalar. The derivative of a scalar with respect to a vector is well defined, it's the vector of individual differentials, i.e. $\frac{\partial z}{\partial x_i}$. Executing for each $x_i$ yields $\frac{\partial z}{\partial x_i}={2x_i \over n}$.

It seems that in your link $x$ is a matrix, and it is $2\times2$, (due to command torch.randn(2,2,...)), not a vector. But, the formulation in the notebook is treating it as if it is a vector, because we're talking about $x_i$'s. However, without loss of generality we can talk about a flattened version of $x$, which is of dimension $4 \times 1$. The mean operation in python takes the mean of each matrix element, so the script is really treating the matrix as if is a vector as we conjecture.

In conclusion, if you let $n=4$, $\frac{2}{n}x$ becomes $x/2$ as it is reported. But, it's a lucky coincidence I believe.

Note: If $x$ is $p\times n$, the derivative is $n\times p$, but this is a minor detail in this calculation.