# How do you derive the gradient for weighted least squares?

So in my previous "adventures in statsland" episode, I believe I was able to convert the weighted sum of squares cost function into matrix form (Formula $\ref{cost}$). $$J(w) = (Xw - y)^T U(Xw-y) \tag{1}\label{cost}$$

Where $X$ is an $m \times n$ input matrix, $w$ is an $n \times 1$ column matrix representing the weights, $y$ is an $m \times 1$ matrix representing your output, and $U$ is an $m \times m$ diagonal matrix where each element $u_{mm}$ weighs the respective input.

Now I am trying to get the gradient of this function with respect to $w$. I've followed the technique outlined in this blog post which derives the gradient for ordinary least squares. However, since $X$ is multiplied by $U$ in our case (the weighing part), the matrices become unwieldy.

Although I'm missing the gradient, I know that the weighted least squares estimage of $w$ is: $$(X^T UX)^{-1} X^T Uy$$

Does anyone know how to derive the gradient or point me to somewhere? I've been searching for hours. Thanks!

$$J (\mathrm w) = (\mathrm X \mathrm w - \mathrm y)^T \mathrm U (\mathrm X \mathrm w - \mathrm y) = \cdots = \mathrm w^T \mathrm X^T \mathrm U \mathrm X \mathrm w - 2 \mathrm w^T \mathrm X^T \mathrm U \mathrm y + \mathrm y^T \mathrm U \mathrm y$$

$$\nabla_{\mathrm w} J (\mathrm w) = 2 \mathrm X^T \mathrm U \mathrm X \mathrm w - 2 \mathrm X^T \mathrm U \mathrm y = 2 \mathrm X^T \mathrm U (\mathrm X \mathrm w - \mathrm y)$$

which vanishes at the solution to the linear system

$$\mathrm X^T \mathrm U \mathrm X \mathrm w = \mathrm X^T \mathrm U \mathrm y$$

If $\mathrm X$ has full column rank and $\mathrm U$ has no zero entries on the main diagonal, the unique solution is

$$\hat{\mathrm w} = (\mathrm X^T \mathrm U \mathrm X)^{-1} \mathrm X^T \mathrm U \mathrm y$$

• Thanks. I have a question about the derivative though. For example, how did $\mathrm w^T \mathrm X^T \mathrm U \mathrm X \mathrm w$ become $2 \mathrm X^T \mathrm U \mathrm X \mathrm w$? – vega Oct 11 '16 at 16:03
• @vega You can write the quadratic form using double summation and then take partial derivatives. It's boring and it's one of those things one does once only. From then on, one consults The Matrix Cookbook in case one forgets about it. – Rodrigo de Azevedo Oct 11 '16 at 20:53
• Ok thanks! Where in the cookbook would I look? – vega Oct 11 '16 at 21:17
• – Rodrigo de Azevedo Oct 11 '16 at 21:19