This question was asked about the gradient of a linear function.

In the answer I don't understand why is $\nabla g(w)=X^t$ (the marked in red part)?

Shouldn't it be just $\nabla g(w)=X$?

And in the part marked in blue, why is $\nabla g(w)$ first written and then $\nabla f(z)$? Shouldn't it be $\nabla f(z) \nabla g(x)$, as order matters in matrices?

I would very much appreciate any clarification.

$$ f(g(x)) = \frac{\partial z}{\partial x}\frac{\partial f}{\partial z}, $$ where $z = g(x)$.

Thus, if we let $$\begin{align} L(w;X,y) = f(y, \text{pred}(X, w)) &:= f(y - \text{pred}(X, w)) \\ &= \|y - \text{pred}(X, w)\|_2^2 \\ & = \|y - Xw\|_2^2, \end{align}$$ then $$ f(z) = \|z\|_2^2 = z^tz $$ and $$ z = g(w) = y - \text{pred}(X, w) = y - Xw. $$ We have, $$ \nabla f(z) = \nabla z^tz = 2z $$ and $$\bbox[11px,border:5px solid #ed1c24]{ \nabla g(w) = \nabla y - Xw = X^t. }$$

Hence, the gradient of your linear estimator is $$ \nabla L(w;x,y) = \bbox[5px,border:5px solid #00a2e8]{\nabla g(w)\nabla f(z)} = X^t\cdot2z = X^t\cdot2(y - Xw). $$


There are two equally valid conventions when writing the derivative of a vector wrt a vector (or a scalar wrt a vector, etc). The first is that $\frac{\partial \vec u}{\partial \vec v}$ is a matrix $M$ such that $M_{ij} = \frac{\partial u_i}{\partial v_j}$. The other convention is $M_{ij} = \frac{\partial u_j}{\partial v_i}$. These are sometimes called numerator vs denominator layout, because the $i$th row of the matrix corresponds to the $i$th element in the numerator/denominator respectively.

The answerer in the other thread is using denominator layout. If you use numerator layout, $\frac{\partial}{\partial w} Xw = X$ and the order of $\nabla f$ and $\nabla g$ will be reversed, due to the chain rule being different depending on the layout.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.