0
$\begingroup$

In the Pytorch Udacity course, the following is said at one point:

To calculate the gradients, you need to run the .backward method on a Variable, z for example. This will calculate the gradient for z with respect to x

Following that, it is said in the course material that

$$ \frac{\partial z}{\partial x} = \frac{\partial}{\partial x}\left[\frac{1}{n}\sum_i^n x_i^2\right] = \frac{x}{2} $$

I would have assumed that one could only take the derivative with respect to an individual $x_i$ since each of them is a separate random variable. However, then the solution would not make sense. I'd be very thankful for some clarity on how to arrive at $x/2$ in this case.

$\endgroup$
  • 2
    $\begingroup$ it seems wrong, it should be 2x/n; how did you get to this? $\endgroup$ – gunes May 29 at 11:14
  • $\begingroup$ It is not my solution, but part of the course material $\endgroup$ – Dan May 29 at 11:16
  • $\begingroup$ I have added a link to the source in my question, the relevant part is in the lower part of the notebook. $\endgroup$ – Dan May 29 at 11:32
1
$\begingroup$

If $x$ is a vector, $z$ is a scalar. The derivative of a scalar with respect to a vector is well defined, it's the vector of individual differentials, i.e. $\frac{\partial z}{\partial x_i}$. Executing for each $x_i$ yields $\frac{\partial z}{\partial x_i}={2x_i \over n}$.

It seems that in your link $x$ is a matrix, and it is $2\times2$, (due to command torch.randn(2,2,...)), not a vector. But, the formulation in the notebook is treating it as if it is a vector, because we're talking about $x_i$'s. However, without loss of generality we can talk about a flattened version of $x$, which is of dimension $4 \times 1$. The mean operation in python takes the mean of each matrix element, so the script is really treating the matrix as if is a vector as we conjecture.

In conclusion, if you let $n=4$, $\frac{2}{n}x$ becomes $x/2$ as it is reported. But, it's a lucky coincidence I believe.

Note: If $x$ is $p\times n$, the derivative is $n\times p$, but this is a minor detail in this calculation.

$\endgroup$
  • $\begingroup$ Thank you! Then it also makes sense that the print(x.grad) command coincides with the reported solution. $\endgroup$ – Dan May 29 at 12:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.