Derivation of MLE weight vector under conditional variance

Question

Suppose we have a data set $\lbrace \mathbf{x}_i, y_i, \sigma_{\mathbf{x}_i}\rbrace_{i=1}^n$ and each $\mathbf{x}_i \in \Bbb{R}^d$

Suppose I want to solve the following for the following:

$$\hat{\mathbf{w}}=\max_{\mathbf{w}} \sum_{i=1}^n \bigg[-\frac{(y_i-\mathbf{x}_i\cdot\mathbf{w})^2}{2\sigma_{\mathbf{x}_i}^2}-\ln\sigma_{\mathbf{x}_i}\sqrt{2\pi}\bigg] $$

I want to solve this. I have made an attempt but I think, due to lack of knowledge, I am approaching it wrong. I would like pointers on how to do it better.

My question is is there a way I can rewrite the derivation I did in an easier matrix notation so that it will make it possible for me to solve with simple matrix operations?

Step 1: I took the gradient

\begin{align*} \mathbf{0} &= \nabla_{\mathbf{w}}\sum_{i=1}^n \bigg[-\frac{(y_i-\mathbf{x}_i\cdot\mathbf{w})^2}{2\sigma_{\mathbf{x}_i}^2}-\ln\sigma_{\mathbf{x}_i}\sqrt{2\pi}\bigg]\\ &= \sum_{i=1}^n \bigg\langle\frac{x_i^1 (y_i-\mathbf{x}_i\cdot\mathbf{w})}{\sigma_{\mathbf{x}_i}^2},...,\frac{x_i^d (y_i-\mathbf{x}_i\cdot\mathbf{w})}{\sigma_{\mathbf{x}_i}^2}\bigg \rangle \end{align*}

Step 2: I tried to simplify

I realized I could write the previous expression as

\begin{align*} \mathbf{0} &= \begin{bmatrix} \frac{(y_1-\mathbf{x}_1\cdot\mathbf{w})}{\sigma_{\mathbf{x}_1}^2}& 0 & \cdots & 0\\ 0 & \frac{(y_2-\mathbf{x}_2\cdot\mathbf{w})}{\sigma_{\mathbf{x}_2}^2} & \cdots & 0\\ \vdots & \vdots & \ddots & \vdots\\ 0 & 0 & \cdots & \frac{(y_n-\mathbf{x}_n\cdot\mathbf{w})}{\sigma_{\mathbf{x}_n}^2}\\ \end{bmatrix} \begin{bmatrix} x_{11} & x_{12} & \cdots & x_{1d}\\ x_{21} & x_{22} & \cdots & x_{2d}\\ \vdots & \vdots & \ddots & \vdots\\ x_{n1} & x_{n2} & \cdots & x_{nd}\\ \end{bmatrix} \begin{bmatrix} 1\\ 1\\ \vdots\\ 1 \end{bmatrix}\\ &= \Sigma^{-1} \begin{bmatrix} (y_1-\mathbf{x}_1\cdot\mathbf{w})& 0 & \cdots & 0\\ 0 & (y_2-\mathbf{x}_2\cdot\mathbf{w}) & \cdots & 0\\ \vdots & \vdots & \ddots & \vdots\\ 0 & 0 & \cdots & (y_n-\mathbf{x}_n\cdot\mathbf{w})\\ \end{bmatrix} \mathbf{X} \begin{bmatrix} 1\\ 1\\ \vdots\\ 1 \end{bmatrix} \end{align*}

I feel there should be an easy set of matrix operations to then obtain $\hat{\mathbf{w}}$, but because of how I've organized the derivation, I think I'm having difficulty simplifying.

Answer 1

Let $Y =\begin{bmatrix} y_1/\sigma_{x_1}\\ y_2/\sigma_{x_2}\\ \vdots\\ y_n/\sigma_{x_n} \end{bmatrix}\\ $, $X=\begin{bmatrix} x_1/\sigma_{x_1}\\ x_2/\sigma_{x_2}\\ \vdots\\ x_n/\sigma_{x_n} \end{bmatrix}\\ $

Then your question can be written as $$\hat{w}=\min_{w}\left((Y-Xw)'(Y-Xw)\right) =\min_{w}\left(Y'Y -2w'X'Y + w'X'Xw\right)$$ Let $L = Y'Y -2w'X'Y + w'X'Xw$, we have $$\frac{dL}{dw}=-2X'Y+X'Xw $$ Let the derivative vector be 0, we get

$$\hat w = (X'X)^{-1}X'Y$$