# Backpropagation gradient of the average

In the Pytorch Udacity course, the following is said at one point:

To calculate the gradients, you need to run the .backward method on a Variable, z for example. This will calculate the gradient for z with respect to x

Following that, it is said in the course material that

$$\frac{\partial z}{\partial x} = \frac{\partial}{\partial x}\left[\frac{1}{n}\sum_i^n x_i^2\right] = \frac{x}{2}$$

I would have assumed that one could only take the derivative with respect to an individual $$x_i$$ since each of them is a separate random variable. However, then the solution would not make sense. I'd be very thankful for some clarity on how to arrive at $$x/2$$ in this case.

• it seems wrong, it should be 2x/n; how did you get to this? – gunes May 29 '19 at 11:14
• It is not my solution, but part of the course material – Dan May 29 '19 at 11:16
• I have added a link to the source in my question, the relevant part is in the lower part of the notebook. – Dan May 29 '19 at 11:32

If $$x$$ is a vector, $$z$$ is a scalar. The derivative of a scalar with respect to a vector is well defined, it's the vector of individual differentials, i.e. $$\frac{\partial z}{\partial x_i}$$. Executing for each $$x_i$$ yields $$\frac{\partial z}{\partial x_i}={2x_i \over n}$$.
It seems that in your link $$x$$ is a matrix, and it is $$2\times2$$, (due to command torch.randn(2,2,...)), not a vector. But, the formulation in the notebook is treating it as if it is a vector, because we're talking about $$x_i$$'s. However, without loss of generality we can talk about a flattened version of $$x$$, which is of dimension $$4 \times 1$$. The mean operation in python takes the mean of each matrix element, so the script is really treating the matrix as if is a vector as we conjecture.
In conclusion, if you let $$n=4$$, $$\frac{2}{n}x$$ becomes $$x/2$$ as it is reported. But, it's a lucky coincidence I believe.
Note: If $$x$$ is $$p\times n$$, the derivative is $$n\times p$$, but this is a minor detail in this calculation.
• Thank you! Then it also makes sense that the print(x.grad) command coincides with the reported solution. – Dan May 29 '19 at 12:06