# Review: Gradient of loss function

In the answer I don't understand why is $\nabla g(w)=X^t$ (the marked in red part)?

Shouldn't it be just $\nabla g(w)=X$?

And in the part marked in blue, why is $\nabla g(w)$ first written and then $\nabla f(z)$? Shouldn't it be $\nabla f(z) \nabla g(x)$, as order matters in matrices?

I would very much appreciate any clarification.

$$f(g(x)) = \frac{\partial z}{\partial x}\frac{\partial f}{\partial z},$$ where $z = g(x)$.

Thus, if we let \begin{align} L(w;X,y) = f(y, \text{pred}(X, w)) &:= f(y - \text{pred}(X, w)) \\ &= \|y - \text{pred}(X, w)\|_2^2 \\ & = \|y - Xw\|_2^2, \end{align} then $$f(z) = \|z\|_2^2 = z^tz$$ and $$z = g(w) = y - \text{pred}(X, w) = y - Xw.$$ We have, $$\nabla f(z) = \nabla z^tz = 2z$$ and $$\bbox[11px,border:5px solid #ed1c24]{ \nabla g(w) = \nabla y - Xw = X^t. }$$

Hence, the gradient of your linear estimator is $$\nabla L(w;x,y) = \bbox[5px,border:5px solid #00a2e8]{\nabla g(w)\nabla f(z)} = X^t\cdot2z = X^t\cdot2(y - Xw).$$

There are two equally valid conventions when writing the derivative of a vector wrt a vector (or a scalar wrt a vector, etc). The first is that $\frac{\partial \vec u}{\partial \vec v}$ is a matrix $M$ such that $M_{ij} = \frac{\partial u_i}{\partial v_j}$. The other convention is $M_{ij} = \frac{\partial u_j}{\partial v_i}$. These are sometimes called numerator vs denominator layout, because the $i$th row of the matrix corresponds to the $i$th element in the numerator/denominator respectively.
The answerer in the other thread is using denominator layout. If you use numerator layout, $\frac{\partial}{\partial w} Xw = X$ and the order of $\nabla f$ and $\nabla g$ will be reversed, due to the chain rule being different depending on the layout.