# Derivation of MLE weight vector under conditional variance

Suppose we have a data set $$\lbrace \mathbf{x}_i, y_i, \sigma_{\mathbf{x}_i}\rbrace_{i=1}^n$$ and each $$\mathbf{x}_i \in \Bbb{R}^d$$

Suppose I want to solve the following for the following:

$$\hat{\mathbf{w}}=\max_{\mathbf{w}} \sum_{i=1}^n \bigg[-\frac{(y_i-\mathbf{x}_i\cdot\mathbf{w})^2}{2\sigma_{\mathbf{x}_i}^2}-\ln\sigma_{\mathbf{x}_i}\sqrt{2\pi}\bigg]$$

I want to solve this. I have made an attempt but I think, due to lack of knowledge, I am approaching it wrong. I would like pointers on how to do it better.

My question is is there a way I can rewrite the derivation I did in an easier matrix notation so that it will make it possible for me to solve with simple matrix operations?

Step 1: I took the gradient

\begin{align*} \mathbf{0} &= \nabla_{\mathbf{w}}\sum_{i=1}^n \bigg[-\frac{(y_i-\mathbf{x}_i\cdot\mathbf{w})^2}{2\sigma_{\mathbf{x}_i}^2}-\ln\sigma_{\mathbf{x}_i}\sqrt{2\pi}\bigg]\\ &= \sum_{i=1}^n \bigg\langle\frac{x_i^1 (y_i-\mathbf{x}_i\cdot\mathbf{w})}{\sigma_{\mathbf{x}_i}^2},...,\frac{x_i^d (y_i-\mathbf{x}_i\cdot\mathbf{w})}{\sigma_{\mathbf{x}_i}^2}\bigg \rangle \end{align*}

Step 2: I tried to simplify

I realized I could write the previous expression as

\begin{align*} \mathbf{0} &= \begin{bmatrix} \frac{(y_1-\mathbf{x}_1\cdot\mathbf{w})}{\sigma_{\mathbf{x}_1}^2}& 0 & \cdots & 0\\ 0 & \frac{(y_2-\mathbf{x}_2\cdot\mathbf{w})}{\sigma_{\mathbf{x}_2}^2} & \cdots & 0\\ \vdots & \vdots & \ddots & \vdots\\ 0 & 0 & \cdots & \frac{(y_n-\mathbf{x}_n\cdot\mathbf{w})}{\sigma_{\mathbf{x}_n}^2}\\ \end{bmatrix} \begin{bmatrix} x_{11} & x_{12} & \cdots & x_{1d}\\ x_{21} & x_{22} & \cdots & x_{2d}\\ \vdots & \vdots & \ddots & \vdots\\ x_{n1} & x_{n2} & \cdots & x_{nd}\\ \end{bmatrix} \begin{bmatrix} 1\\ 1\\ \vdots\\ 1 \end{bmatrix}\\ &= \Sigma^{-1} \begin{bmatrix} (y_1-\mathbf{x}_1\cdot\mathbf{w})& 0 & \cdots & 0\\ 0 & (y_2-\mathbf{x}_2\cdot\mathbf{w}) & \cdots & 0\\ \vdots & \vdots & \ddots & \vdots\\ 0 & 0 & \cdots & (y_n-\mathbf{x}_n\cdot\mathbf{w})\\ \end{bmatrix} \mathbf{X} \begin{bmatrix} 1\\ 1\\ \vdots\\ 1 \end{bmatrix} \end{align*}

I feel there should be an easy set of matrix operations to then obtain $$\hat{\mathbf{w}}$$, but because of how I've organized the derivation, I think I'm having difficulty simplifying.

Let $$Y =\begin{bmatrix} y_1/\sigma_{x_1}\\ y_2/\sigma_{x_2}\\ \vdots\\ y_n/\sigma_{x_n} \end{bmatrix}\\$$, $$X=\begin{bmatrix} x_1/\sigma_{x_1}\\ x_2/\sigma_{x_2}\\ \vdots\\ x_n/\sigma_{x_n} \end{bmatrix}\\$$
Then your question can be written as $$\hat{w}=\min_{w}\left((Y-Xw)'(Y-Xw)\right) =\min_{w}\left(Y'Y -2w'X'Y + w'X'Xw\right)$$ Let $$L = Y'Y -2w'X'Y + w'X'Xw$$, we have $$\frac{dL}{dw}=-2X'Y+X'Xw$$ Let the derivative vector be 0, we get
$$\hat w = (X'X)^{-1}X'Y$$
• Assume your conditional variance means $\sigma_i^2$. The property of OLS is BLUE (best linear unbiased estimate) when the assumptions are met. If homoscedasticity is not true ($\sigma_i^2 \ne \sigma_i'^2$), then OLS is not BLUE, although it is unbiased. After transformation, the random vector Y has the property of homogeneity. It is WLS (weighted least square). I assume $\sigma_i^2$ is known constant. – user158565 Oct 13 '18 at 17:59