I am trying to understand the origin of the weighted least squares estimation. I'll try to describe my thought process:

  1. Introduction

    Let's say that we have the following system: $$ y = Hx + v, $$ where $x \in \mathbb{R}^n$ is some constant vector that we want to estimate, $y \in \mathbb{R}^k$ is a vector of $k$ measurements, $H \in \mathbb{R}^{k \times n}$ is a linear transformation matrix, and $v$ is a vector of $k$ zero-mean independent random variables (let's say something like a measurement noise). Therefore, the covariance matrix of $v$ is defined as: $$ \mathbb{E}[v v^T] = R = \left[ \begin{array}{cccc} \sigma_1^2 & 0 & \dots & 0 \\ 0 & \sigma_2^2 & \dots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \dots & \sigma_k^2 \end{array}\right] , \qquad \mathbb{E}[v_i v_j] = \left\{ \begin{array}{ll} \sigma_i^2 , & i=j, \\ 0 , & i \neq j, \end{array} \right. $$ where $v_i$ is the $i$-th random variable of vector $v$.

  2. Ordinary least squares estimation

    Consider the system without measurement noise, i.e., $y=Hx$. In that case, an objective function that will be used to determine an estimate of $x$ is defined as: $$ J(x) = e^T e = (y-Hx)^T (y-Hx) = x^T H^T H x - 2 y^T H x + y^T y $$ The best estimate $\hat{x}$ considering the objective function $J(x)$ is defined as $\hat{x} = \arg \min_{x} J(x)$. The partial derivative of $J(x)$ with respect to $x$ at $\hat{x}$ is equal to $0$, i.e., $$ \left. \frac{\partial J(x)}{\partial x} \right|_{x=\hat{x}} = 2 H^T H x - 2 H^T y = 0 $$ Finally, it follows that the best estimate $\hat{x}$ considering objective $J(x)$ is defined as: $$ \hat{x} = (H^T H)^{-1} H^T y $$ It is easily verified that $\hat{x}$ is indeed a minimizer of $J(x)$, since $$ \left. \frac{\partial^2 J(x)}{\partial x^2} \right|_{x=\hat{x}} = 2 H^T H \geq 0 $$

  3. Least squares estimation with constant $\sigma$

    Now, the same expression for optimal estimate $\hat{x}$ is obtained if we also include the measurement noise $v$, and all variances $\sigma_i^2$ are equal. In that case, the covariance matrix $R$ is $$ R = \sigma^2 I_{k} , \qquad \mathbb{E}[v_1^2] = \ldots = \mathbb{E}[v_k^2] = \sigma^2 $$ where $I_k$ is an $k \times k$ identity matrix. Although this is intuitively clear, I'm having troubles with mathematical formulation of this assumption, i.e., I don't know how to repeat steps as explained in Section 2, e.g., I don't know what should I use as $J(x)$.

  4. Weighted least squares estimation

    And finally we consider the system $$ y = Hx + v $$ as described in Section 1. The covariance matrix of $v$, denoted as $R$, is still diagonal, but now variances $\sigma_i^2$ are different. Now we introduce a new measurement noise vector as follows: $$ w = R^{-\frac{1}{2}} v, \qquad \mathbb{E}[w w^T] = I_k $$ where $I_k$ is an $k \times k$ identity matrix. If we now write the starting system as: $$ R^{-\frac{1}{2}} y = R^{-\frac{1}{2}} H x + w $$ In other words, we get the same system as described in Section 3, i.e., the covariance matrix of $w$ is an identity matrix.

  5. Conclusion

    Can someone help me with a mathematical formulation of LSE when there is some noise vector included, as asked in Section 3?

  • $\begingroup$ Is this a question from a course or textbook? If so, please add the [self-study] tag & read its wiki. $\endgroup$ – gung - Reinstate Monica Dec 1 '15 at 21:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.