Suppose we have a data set $\lbrace \mathbf{x}_i, y_i, \sigma_{\mathbf{x}_i}\rbrace_{i=1}^n$ and each $\mathbf{x}_i \in \Bbb{R}^d$

Suppose I want to solve the following for the following:

$$\hat{\mathbf{w}}=\max_{\mathbf{w}} \sum_{i=1}^n \bigg[-\frac{(y_i-\mathbf{x}_i\cdot\mathbf{w})^2}{2\sigma_{\mathbf{x}_i}^2}-\ln\sigma_{\mathbf{x}_i}\sqrt{2\pi}\bigg] $$

I want to solve this. I have made an attempt but I think, due to lack of knowledge, I am approaching it wrong. I would like pointers on how to do it better.

My question is is there a way I can rewrite the derivation I did in an easier matrix notation so that it will make it possible for me to solve with simple matrix operations?

Step 1: I took the gradient

\begin{align*} \mathbf{0} &= \nabla_{\mathbf{w}}\sum_{i=1}^n \bigg[-\frac{(y_i-\mathbf{x}_i\cdot\mathbf{w})^2}{2\sigma_{\mathbf{x}_i}^2}-\ln\sigma_{\mathbf{x}_i}\sqrt{2\pi}\bigg]\\ &= \sum_{i=1}^n \bigg\langle\frac{x_i^1 (y_i-\mathbf{x}_i\cdot\mathbf{w})}{\sigma_{\mathbf{x}_i}^2},...,\frac{x_i^d (y_i-\mathbf{x}_i\cdot\mathbf{w})}{\sigma_{\mathbf{x}_i}^2}\bigg \rangle \end{align*}

Step 2: I tried to simplify

I realized I could write the previous expression as

\begin{align*} \mathbf{0} &= \begin{bmatrix} \frac{(y_1-\mathbf{x}_1\cdot\mathbf{w})}{\sigma_{\mathbf{x}_1}^2}& 0 & \cdots & 0\\ 0 & \frac{(y_2-\mathbf{x}_2\cdot\mathbf{w})}{\sigma_{\mathbf{x}_2}^2} & \cdots & 0\\ \vdots & \vdots & \ddots & \vdots\\ 0 & 0 & \cdots & \frac{(y_n-\mathbf{x}_n\cdot\mathbf{w})}{\sigma_{\mathbf{x}_n}^2}\\ \end{bmatrix} \begin{bmatrix} x_{11} & x_{12} & \cdots & x_{1d}\\ x_{21} & x_{22} & \cdots & x_{2d}\\ \vdots & \vdots & \ddots & \vdots\\ x_{n1} & x_{n2} & \cdots & x_{nd}\\ \end{bmatrix} \begin{bmatrix} 1\\ 1\\ \vdots\\ 1 \end{bmatrix}\\ &= \Sigma^{-1} \begin{bmatrix} (y_1-\mathbf{x}_1\cdot\mathbf{w})& 0 & \cdots & 0\\ 0 & (y_2-\mathbf{x}_2\cdot\mathbf{w}) & \cdots & 0\\ \vdots & \vdots & \ddots & \vdots\\ 0 & 0 & \cdots & (y_n-\mathbf{x}_n\cdot\mathbf{w})\\ \end{bmatrix} \mathbf{X} \begin{bmatrix} 1\\ 1\\ \vdots\\ 1 \end{bmatrix} \end{align*}

I feel there should be an easy set of matrix operations to then obtain $\hat{\mathbf{w}}$, but because of how I've organized the derivation, I think I'm having difficulty simplifying.


Let $Y =\begin{bmatrix} y_1/\sigma_{x_1}\\ y_2/\sigma_{x_2}\\ \vdots\\ y_n/\sigma_{x_n} \end{bmatrix}\\ $, $X=\begin{bmatrix} x_1/\sigma_{x_1}\\ x_2/\sigma_{x_2}\\ \vdots\\ x_n/\sigma_{x_n} \end{bmatrix}\\ $

Then your question can be written as $$\hat{w}=\min_{w}\left((Y-Xw)'(Y-Xw)\right) =\min_{w}\left(Y'Y -2w'X'Y + w'X'Xw\right)$$ Let $L = Y'Y -2w'X'Y + w'X'Xw$, we have $$\frac{dL}{dw}=-2X'Y+X'Xw $$ Let the derivative vector be 0, we get

$$\hat w = (X'X)^{-1}X'Y$$

| cite | improve this answer | |
  • $\begingroup$ Why doesn't the conditional variance make OLS inapplicable? I thought one of the assumptions for OLS was that you had to have homoscedasticity? Are you effectively eliminating that issue by dividing by the variances? $\endgroup$ – Stan Shunpike Oct 13 '18 at 17:31
  • 1
    $\begingroup$ Assume your conditional variance means $\sigma_i^2$. The property of OLS is BLUE (best linear unbiased estimate) when the assumptions are met. If homoscedasticity is not true ($\sigma_i^2 \ne \sigma_i'^2$), then OLS is not BLUE, although it is unbiased. After transformation, the random vector Y has the property of homogeneity. It is WLS (weighted least square). I assume $\sigma_i^2$ is known constant. $\endgroup$ – user158565 Oct 13 '18 at 17:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.