So in my previous "adventures in statsland" episode, I believe I was able to convert the weighted sum of squares cost function into matrix form (Formula $\ref{cost}$). $$ J(w) = (Xw - y)^T U(Xw-y) \tag{1}\label{cost} $$

Where $X$ is an $m \times n$ input matrix, $w$ is an $n \times 1$ column matrix representing the weights, $y$ is an $m \times 1$ matrix representing your output, and $U$ is an $m \times m$ diagonal matrix where each element $u_{mm}$ weighs the respective input.

Now I am trying to get the gradient of this function with respect to $w$. I've followed the technique outlined in this blog post which derives the gradient for ordinary least squares. However, since $X$ is multiplied by $U$ in our case (the weighing part), the matrices become unwieldy.

Although I'm missing the gradient, I know that the weighted least squares estimage of $w$ is: $$ (X^T UX)^{-1} X^T Uy $$

Does anyone know how to derive the gradient or point me to somewhere? I've been searching for hours. Thanks!


$$J (\mathrm w) = (\mathrm X \mathrm w - \mathrm y)^T \mathrm U (\mathrm X \mathrm w - \mathrm y) = \cdots = \mathrm w^T \mathrm X^T \mathrm U \mathrm X \mathrm w - 2 \mathrm w^T \mathrm X^T \mathrm U \mathrm y + \mathrm y^T \mathrm U \mathrm y$$

Taking the gradient,

$$\nabla_{\mathrm w} J (\mathrm w) = 2 \mathrm X^T \mathrm U \mathrm X \mathrm w - 2 \mathrm X^T \mathrm U \mathrm y = 2 \mathrm X^T \mathrm U (\mathrm X \mathrm w - \mathrm y)$$

which vanishes at the solution to the linear system

$$\mathrm X^T \mathrm U \mathrm X \mathrm w = \mathrm X^T \mathrm U \mathrm y$$

If $\mathrm X$ has full column rank and $\mathrm U$ has no zero entries on the main diagonal, the unique solution is

$$\hat{\mathrm w} = (\mathrm X^T \mathrm U \mathrm X)^{-1} \mathrm X^T \mathrm U \mathrm y$$

  • $\begingroup$ Thanks. I have a question about the derivative though. For example, how did $ \mathrm w^T \mathrm X^T \mathrm U \mathrm X \mathrm w$ become $2 \mathrm X^T \mathrm U \mathrm X \mathrm w$? $\endgroup$ – vega Oct 11 '16 at 16:03
  • 1
    $\begingroup$ @vega You can write the quadratic form using double summation and then take partial derivatives. It's boring and it's one of those things one does once only. From then on, one consults The Matrix Cookbook in case one forgets about it. $\endgroup$ – Rodrigo de Azevedo Oct 11 '16 at 20:53
  • $\begingroup$ Ok thanks! Where in the cookbook would I look? $\endgroup$ – vega Oct 11 '16 at 21:17
  • $\begingroup$ @vega Page 12, section 2.4.4 $\endgroup$ – Rodrigo de Azevedo Oct 11 '16 at 21:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.