# Derivation of the weighted least squares (WLS) method

I am trying to understand the origin of the weighted least squares estimation. I'll try to describe my thought process:

1. Introduction

Let's say that we have the following system: $$y = Hx + v,$$ where $x \in \mathbb{R}^n$ is some constant vector that we want to estimate, $y \in \mathbb{R}^k$ is a vector of $k$ measurements, $H \in \mathbb{R}^{k \times n}$ is a linear transformation matrix, and $v$ is a vector of $k$ zero-mean independent random variables (let's say something like a measurement noise). Therefore, the covariance matrix of $v$ is defined as: $$\mathbb{E}[v v^T] = R = \left[ \begin{array}{cccc} \sigma_1^2 & 0 & \dots & 0 \\ 0 & \sigma_2^2 & \dots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \dots & \sigma_k^2 \end{array}\right] , \qquad \mathbb{E}[v_i v_j] = \left\{ \begin{array}{ll} \sigma_i^2 , & i=j, \\ 0 , & i \neq j, \end{array} \right.$$ where $v_i$ is the $i$-th random variable of vector $v$.

2. Ordinary least squares estimation

Consider the system without measurement noise, i.e., $y=Hx$. In that case, an objective function that will be used to determine an estimate of $x$ is defined as: $$J(x) = e^T e = (y-Hx)^T (y-Hx) = x^T H^T H x - 2 y^T H x + y^T y$$ The best estimate $\hat{x}$ considering the objective function $J(x)$ is defined as $\hat{x} = \arg \min_{x} J(x)$. The partial derivative of $J(x)$ with respect to $x$ at $\hat{x}$ is equal to $0$, i.e., $$\left. \frac{\partial J(x)}{\partial x} \right|_{x=\hat{x}} = 2 H^T H x - 2 H^T y = 0$$ Finally, it follows that the best estimate $\hat{x}$ considering objective $J(x)$ is defined as: $$\hat{x} = (H^T H)^{-1} H^T y$$ It is easily verified that $\hat{x}$ is indeed a minimizer of $J(x)$, since $$\left. \frac{\partial^2 J(x)}{\partial x^2} \right|_{x=\hat{x}} = 2 H^T H \geq 0$$

3. Least squares estimation with constant $\sigma$

Now, the same expression for optimal estimate $\hat{x}$ is obtained if we also include the measurement noise $v$, and all variances $\sigma_i^2$ are equal. In that case, the covariance matrix $R$ is $$R = \sigma^2 I_{k} , \qquad \mathbb{E}[v_1^2] = \ldots = \mathbb{E}[v_k^2] = \sigma^2$$ where $I_k$ is an $k \times k$ identity matrix. Although this is intuitively clear, I'm having troubles with mathematical formulation of this assumption, i.e., I don't know how to repeat steps as explained in Section 2, e.g., I don't know what should I use as $J(x)$.

4. Weighted least squares estimation

And finally we consider the system $$y = Hx + v$$ as described in Section 1. The covariance matrix of $v$, denoted as $R$, is still diagonal, but now variances $\sigma_i^2$ are different. Now we introduce a new measurement noise vector as follows: $$w = R^{-\frac{1}{2}} v, \qquad \mathbb{E}[w w^T] = I_k$$ where $I_k$ is an $k \times k$ identity matrix. If we now write the starting system as: $$R^{-\frac{1}{2}} y = R^{-\frac{1}{2}} H x + w$$ In other words, we get the same system as described in Section 3, i.e., the covariance matrix of $w$ is an identity matrix.

5. Conclusion

Can someone help me with a mathematical formulation of LSE when there is some noise vector included, as asked in Section 3?

• Is this a question from a course or textbook? If so, please add the [self-study] tag & read its wiki. – gung - Reinstate Monica Dec 1 '15 at 21:11