# Differentiating the RSS w.r.t. $\beta$ in Linear Model

I am reading the book "The Elements of Statistical Learning". The book says

But when I try to prove it, I get the following:

$$RSS(\beta) = (y - X\beta)^T(y-X\beta)$$ $$RSS(\beta) = y^Ty -\beta^TX^Ty -y^TX\beta+\beta^TX^TX\beta$$ $$\frac{\partial{RSS}}{\partial{\beta}} = -y^TX - y^TX + \beta^T(X^TX + X^TX) = -2\beta^T(y^TX +X^TX)$$

What is wrong in my derivation ？

I see the wiki page about the Matrix Calculus. I find I was misled by the meaning about the Numerator-layout notation or Denominator-layout notation. Can anyone give an intuitive explanation when use Numerator-layout notation and when use Denominator-layout notation.

• I just think of it like, (y-XB)^2, so the derivative of this is -2X(y-XB), and derivative of that is 2X^2 Apr 9, 2016 at 4:55
• Hi，I know what you means, but just feel confused about the dimensions when we differentiate a scalar by a vector ?
– 陈家泽
Apr 9, 2016 at 5:47
• This concept is covered in textbooks on multivariable calculus, which begin by discussing real-valued functions of multiple real values, define their derivative, and then relate that derivative to the partial derivatives. Thus, referring to your favorite textbook ought to clear up any confusion.
– whuber
Apr 9, 2016 at 18:11

Note that $-y^TX - y^TX + \beta^T(X^TX + X^TX) \ne -2\beta^T(y^TX + X^TX)$. You pulled an extra $\beta^T$ out from the first term. So you were wrong with the algebra there.

You have, $$RSS = y^Ty - \beta^TX^Ty - y^TX \beta + \beta^TX^TX\beta.$$ Notice that $-y^TX \beta$ is a scalar quantity, thus $y^TX\beta = (y^TX \beta)^T = \beta^TX^Ty$. $$\frac{\partial{RSS}}{\partial{\beta}} = -2X^Ty + 2X^TX\beta.$$

\begin{align*} -2X^Ty + 2X^TX \beta & = -2X^T(y - X\beta). \end{align*}

• I'm trying to understand how $-y^TX$ is a scalar quantity. To me it looks like a horizontal vector. If $n$ is the number of observations and $k$ is the number of explanatory variables, then $y$ is a vertical vector length $n$, and $X$ is an $n \times k$ matrix. Then $-y^TX$ would seem to be multiplying an object with dimensions $1 \times n$ by an object with dimensions $n \times k$, thus producing an object dimension $1 \times k$ - a horizontal vector length $k$, no? I'm sure I'm missing something here - the final result is, after all, correct - can you clarify? Thx. Dec 12, 2016 at 5:29
• @sparc_spread You are right. That was a silly oversight. Fixed the answer I think now. Dec 12, 2016 at 6:30