4
$\begingroup$

In deriving the parameter estimate in OLS, we differentiate the following (in matrix form)

$$y^T y - 2\beta^T X^T y + \beta^T X^T X \beta$$

The part of the differentiation I don't understand is why $$\beta^T X^T X \beta$$ differentiates to

$$2X^T X \beta. $$

The way I thought it would work, was:

$$\triangledown(\beta^T)X^TX\beta + \beta^TX^TX\triangledown (\beta) = X^TX \beta + \beta^TX^TX $$

$\endgroup$
6
$\begingroup$

In your way, the dimensions of the summands don't match. If you write the term as $\beta^TA\beta$ openly, we'll have the following expression ($A$ is symmetric): $$\beta^TA\beta=\sum_{i,j}A_{ij}\beta_i\beta_j=\sum_i A_{ii}\beta_i^2+2\sum_{i<j}A_{ij}\beta_i\beta_j$$ Derivative with respect to $\beta_i$ is $$\frac{\partial \beta^TA\beta}{\partial \beta_i}=2\beta_iA_{ii}+2A_{ij}\beta_j$$ This expression is basically the dot product of $i$-th row of $A$ and $\beta$ multiplied by $2$, i.e. $2A_i^T\beta$, where $A_i$ denotes the $i$-th row.

In numerator layout notation, which is commonly used in matrix calculus cheatsheets, scalar differentiated by a vector produces a horizontal vector, so we'll concatenate the derivative for each $\beta_i$ horizontally:

$$\frac{\partial\beta^TA\beta}{\partial\beta}=[2A_1^T\beta\dots2A_n^T\beta]=2[A_1^T\dots A_n^T]\beta$$

If we transpose each row of A and concatenate them horizontally in their respective order, we obtain $A^T$, which is $A$ since $A$ was symmetric. Therefore, the middle matrix in the above expression is $A$. Letting $A=X^TX$, we have:

$$\frac{\partial\beta^TA\beta}{\partial\beta}=2A^T\beta=2A\beta=2X^TX\beta$$

which matches your formula.

$\endgroup$
2
  • $\begingroup$ How can I look at it from simply a matrix point of view? $\endgroup$
    – Bill
    Jan 23 at 17:37
  • 1
    $\begingroup$ For a rigorous account of matrix differentiation, see stats.stackexchange.com/a/257616/919. But as a practical matter, dimensional considerations like those used in the present answer are a powerful way to avoid mistakes. $\endgroup$
    – whuber
    Jan 23 at 18:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.