2
$\begingroup$

So in my previous "adventures in statsland" episode, I believe I was able to convert the weighted sum of squares cost function into matrix form (Formula $\ref{cost}$). $$ J(w) = (Xw - y)^T U(Xw-y) \tag{1}\label{cost} $$

Where $X$ is an $m \times n$ input matrix, $w$ is an $n \times 1$ column matrix representing the weights, $y$ is an $m \times 1$ matrix representing your output, and $U$ is an $m \times m$ diagonal matrix where each element $u_{mm}$ weighs the respective input.

Now I am trying to get the gradient of this function with respect to $w$. I've followed the technique outlined in this blog post which derives the gradient for ordinary least squares. However, since $X$ is multiplied by $U$ in our case (the weighing part), the matrices become unwieldy.

Although I'm missing the gradient, I know that the weighted least squares estimage of $w$ is: $$ (X^T UX)^{-1} X^T Uy $$

Does anyone know how to derive the gradient or point me to somewhere? I've been searching for hours. Thanks!

$\endgroup$

1 Answer 1

1
$\begingroup$

$$J (\mathrm w) = (\mathrm X \mathrm w - \mathrm y)^T \mathrm U (\mathrm X \mathrm w - \mathrm y) = \cdots = \mathrm w^T \mathrm X^T \mathrm U \mathrm X \mathrm w - 2 \mathrm w^T \mathrm X^T \mathrm U \mathrm y + \mathrm y^T \mathrm U \mathrm y$$

Taking the gradient,

$$\nabla_{\mathrm w} J (\mathrm w) = 2 \mathrm X^T \mathrm U \mathrm X \mathrm w - 2 \mathrm X^T \mathrm U \mathrm y = 2 \mathrm X^T \mathrm U (\mathrm X \mathrm w - \mathrm y)$$

which vanishes at the solution to the linear system

$$\mathrm X^T \mathrm U \mathrm X \mathrm w = \mathrm X^T \mathrm U \mathrm y$$

If $\mathrm X$ has full column rank and $\mathrm U$ has no zero entries on the main diagonal, the unique solution is

$$\hat{\mathrm w} = (\mathrm X^T \mathrm U \mathrm X)^{-1} \mathrm X^T \mathrm U \mathrm y$$

$\endgroup$
4
  • $\begingroup$ Thanks. I have a question about the derivative though. For example, how did $ \mathrm w^T \mathrm X^T \mathrm U \mathrm X \mathrm w$ become $2 \mathrm X^T \mathrm U \mathrm X \mathrm w$? $\endgroup$
    – vega
    Oct 11, 2016 at 16:03
  • 1
    $\begingroup$ @vega You can write the quadratic form using double summation and then take partial derivatives. It's boring and it's one of those things one does once only. From then on, one consults The Matrix Cookbook in case one forgets about it. $\endgroup$ Oct 11, 2016 at 20:53
  • $\begingroup$ Ok thanks! Where in the cookbook would I look? $\endgroup$
    – vega
    Oct 11, 2016 at 21:17
  • $\begingroup$ @vega Page 12, section 2.4.4 $\endgroup$ Oct 11, 2016 at 21:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.