8
$\begingroup$

I am reading the book "The Elements of Statistical Learning". The book says enter image description here

But when I try to prove it, I get the following:

$$RSS(\beta) = (y - X\beta)^T(y-X\beta)$$ $$RSS(\beta) = y^Ty -\beta^TX^Ty -y^TX\beta+\beta^TX^TX\beta$$ $$\frac{\partial{RSS}}{\partial{\beta}} = -y^TX - y^TX + \beta^T(X^TX + X^TX) = -2\beta^T(y^TX +X^TX)$$

What is wrong in my derivation ?

I see the wiki page about the Matrix Calculus. I find I was misled by the meaning about the Numerator-layout notation or Denominator-layout notation. Can anyone give an intuitive explanation when use Numerator-layout notation and when use Denominator-layout notation.

$\endgroup$
3
  • $\begingroup$ I just think of it like, (y-XB)^2, so the derivative of this is -2X(y-XB), and derivative of that is 2X^2 $\endgroup$
    – conv3d
    Apr 9, 2016 at 4:55
  • $\begingroup$ Hi,I know what you means, but just feel confused about the dimensions when we differentiate a scalar by a vector ? $\endgroup$
    – 陈家泽
    Apr 9, 2016 at 5:47
  • $\begingroup$ This concept is covered in textbooks on multivariable calculus, which begin by discussing real-valued functions of multiple real values, define their derivative, and then relate that derivative to the partial derivatives. Thus, referring to your favorite textbook ought to clear up any confusion. $\endgroup$
    – whuber
    Apr 9, 2016 at 18:11

1 Answer 1

8
$\begingroup$

Note that $-y^TX - y^TX + \beta^T(X^TX + X^TX) \ne -2\beta^T(y^TX + X^TX)$. You pulled an extra $\beta^T$ out from the first term. So you were wrong with the algebra there.

You have, $$RSS = y^Ty - \beta^TX^Ty - y^TX \beta + \beta^TX^TX\beta.$$ Notice that $-y^TX \beta$ is a scalar quantity, thus $y^TX\beta = (y^TX \beta)^T = \beta^TX^Ty$. $$\frac{\partial{RSS}}{\partial{\beta}} = -2X^Ty + 2X^TX\beta.$$

\begin{align*} -2X^Ty + 2X^TX \beta & = -2X^T(y - X\beta). \end{align*}

$\endgroup$
2
  • $\begingroup$ I'm trying to understand how $-y^TX$ is a scalar quantity. To me it looks like a horizontal vector. If $n$ is the number of observations and $k$ is the number of explanatory variables, then $y$ is a vertical vector length $n$, and $X$ is an $n \times k$ matrix. Then $-y^TX$ would seem to be multiplying an object with dimensions $1 \times n$ by an object with dimensions $n \times k$, thus producing an object dimension $1 \times k$ - a horizontal vector length $k$, no? I'm sure I'm missing something here - the final result is, after all, correct - can you clarify? Thx. $\endgroup$ Dec 12, 2016 at 5:29
  • $\begingroup$ @sparc_spread You are right. That was a silly oversight. Fixed the answer I think now. $\endgroup$ Dec 12, 2016 at 6:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.