Differentiating $ (y-X\beta)^T(y - X \beta) $ with respect to $\beta$

Question

How do I differentiate $$ (y-X\beta)^T(y - X \beta) $$

with respect to $\beta$. The result I saw was

$$X^T(y - X\beta)$$

Answer 1

Let us assume that you are working in a setup where $y$ is $N \times 1$ and $X$ is $N \times K$ and $\beta$ is $K \times 1$. I prefer to define $e(\beta) := (y - X\beta)$ and similarly the $i$'th component $e_{i}(\beta) = (y - X\beta)_i = y_i - x_i^\top\beta$ where $x_i^\top$ is the $i$'th row of $X$. You should then be able to convince yourself that

$$e(\beta)^\top e(\beta) = \sum_i e_{i}(\beta) e_{i}(\beta),$$

the sum of squared deviations. Now I guess you know how to differentiate with respect to a single variable (read parameter) $\beta_k$ so lets try this

$$\frac{\partial}{\partial \beta_k} e(\beta)^\top e(\beta) = \sum_i\frac{\partial}{\partial \beta_k} [e_{i}(\beta) e_{i}(\beta)],$$

apply the product rule to get

$$= \sum_i \frac{\partial e_i(\beta)}{\partial \beta_k} e_i(\beta) + e_i(\beta) \frac{\partial e_i(\beta)}{\partial \beta_k} = 2 \sum_i \frac{\partial e_i(\beta)}{\partial \beta_k} e_i(\beta),$$

where the final sum here can be written in matrix/vector notation as

$$= 2 \left[\frac{\partial e(\beta)^\top}{\partial \beta_k}\right] e(\beta),$$

all the same derivations can be done differentiating with respect to a column $\beta$ observing the rule that when you differentiate with respect to a column you get a column so

$$\frac{\partial e_i(\beta)}{\partial \beta} = \begin{pmatrix} \frac{\partial e_i(\beta)}{\partial \beta_1}\\ \vdots \\ \frac{\partial e_i(\beta)}{\partial \beta_K} \end{pmatrix}$$

you should then be able to get the rule that

$$\frac{\partial}{\partial \beta} e(\beta)^\top e(\beta) = 2 \left[\frac{\partial e(\beta)^\top}{\partial \beta}\right] e(\beta),$$

so let figure out what $\frac{\partial e(\beta)^\top}{\partial \beta}$ for which we get

$$\frac{\partial e(\beta)^\top}{\partial \beta} = \frac{\partial}{\partial \beta} (e_1(\beta),...,e_N(\beta)) = \left( \frac{\partial e_1(\beta)}{\partial \beta},..., \frac{\partial e_N(\beta)}{\partial \beta}\right)$$ and for each $i$ you have that $\frac{\partial e_{i}(\beta)}{\partial \beta} = -x_i$ so then it is easy to see that $$\frac{\partial e(\beta)^\top}{\partial \beta} = - X^\top$$ and it follows that

$$\frac{\partial}{\partial \beta} e(\beta)^\top e(\beta) = - 2X^\top (y - X\beta).$$

In a context where the writer knows he or she wants to solve $- 2X^\top (y - X\beta) = 0$ he or she may go directly from $$\frac{\partial}{\partial \beta} e(\beta)^\top e(\beta) = 0$$ to $X^\top (y - X\beta) = 0$ leading you to think that the author is implicitly claiming that $$\frac{\partial}{\partial \beta} e(\beta)^\top e(\beta)= X^\top (y - X\beta) $$ which is not the case.