In this PyTorch tutorial the backprop to compute gradients is shown with the following code:
# Backprop to compute gradients of w1 and w2 with respect to loss grad_y_pred = 2.0 * (y_pred - y) grad_w2 = h_relu.T.dot(grad_y_pred) grad_h_relu = grad_y_pred.dot(w2.T) grad_h = grad_h_relu.copy() grad_h[h < 0] = 0 grad_w1 = x.T.dot(grad_h)
I don't understand the gradient calculation in the above snippet. Can anyone provide a comment on this? E.g. why do we multiple by 2 to get the grad_y_pred?