How do you derive the gradient for weighted least squares?

Question

So in my previous "adventures in statsland" episode, I believe I was able to convert the weighted sum of squares cost function into matrix form (Formula $\ref{cost}$). $$ J(w) = (Xw - y)^T U(Xw-y) \tag{1}\label{cost} $$

Where $X$ is an $m \times n$ input matrix, $w$ is an $n \times 1$ column matrix representing the weights, $y$ is an $m \times 1$ matrix representing your output, and $U$ is an $m \times m$ diagonal matrix where each element $u_{mm}$ weighs the respective input.

Now I am trying to get the gradient of this function with respect to $w$. I've followed the technique outlined in this blog post which derives the gradient for ordinary least squares. However, since $X$ is multiplied by $U$ in our case (the weighing part), the matrices become unwieldy.

Although I'm missing the gradient, I know that the weighted least squares estimage of $w$ is: $$ (X^T UX)^{-1} X^T Uy $$

Does anyone know how to derive the gradient or point me to somewhere? I've been searching for hours. Thanks!

Answer 1

$$J (\mathrm w) = (\mathrm X \mathrm w - \mathrm y)^T \mathrm U (\mathrm X \mathrm w - \mathrm y) = \cdots = \mathrm w^T \mathrm X^T \mathrm U \mathrm X \mathrm w - 2 \mathrm w^T \mathrm X^T \mathrm U \mathrm y + \mathrm y^T \mathrm U \mathrm y$$

Taking the gradient,

$$\nabla_{\mathrm w} J (\mathrm w) = 2 \mathrm X^T \mathrm U \mathrm X \mathrm w - 2 \mathrm X^T \mathrm U \mathrm y = 2 \mathrm X^T \mathrm U (\mathrm X \mathrm w - \mathrm y)$$

which vanishes at the solution to the linear system

$$\mathrm X^T \mathrm U \mathrm X \mathrm w = \mathrm X^T \mathrm U \mathrm y$$

If $\mathrm X$ has full column rank and $\mathrm U$ has no zero entries on the main diagonal, the unique solution is

$$\hat{\mathrm w} = (\mathrm X^T \mathrm U \mathrm X)^{-1} \mathrm X^T \mathrm U \mathrm y$$