Sequential linear estimation: Is MSE non-increasing?

Question

Suppose we estimate $x$ from $y_1=ax+z_1$ and get $x_1$. Then suppose we receive $y_2=bx+z_2$ and reestimate $x$ using both $y_1$ and $y_2$ and get $x_2$. Is the means squared error (MSE) of $x_2$ always lower than MSE of $x_1$. In other words is $$E(x-x_1)^2\geq E(x-x_2)^2?$$ Here $a,b$ are know constants and $z_1$ and $z_2$ are Gaussian errors and the estimation is linear minimum mean squared error. The equations can also be vectors in that case error is the trance of the error covariance matrix. By this wiki it seems to be decreasing.

Edit: explanation

At time $t=0$ a real value $x$ is generated by a random process $X$. Then at time $t=1$, I receive a noise corrupted and scaled version $y_1$ of $x$ denoted by $y_1=ax+z_1$. From $y_1$ I do linear minimum mean squared error (LMMSE) estimation and estimate $x$ and get $x_1$ and the corresponding LMMSE is $E(x-x_1)^2$. Here $a$ is a constant and is known, $z_1$ is Gaussian noise with known mean and variance. LMMSE estimation is straightforward.

Then at time $t=2$, I receive another noise corrupted and scaled version $y_2$ of that same $x$ which is denoted by $y_2=bx+z_2$. Again $b$ is a known constant and $z_2$ is Gaussian noise with known mean and variance. Now I simultaneously use both $y_1$ and $y_2$ to estimate $x$ by LMMSE estimation and I get the value $x_2$ and the corresponding error $E(x-x_2)^2$. My question: is $E(x-x_1)^2\geq E(x-x_2)^2?$. i.e., the time actually does not matter the question is does LMMSE estimation using $n+1$ samples always give a less error than estimating only using $n$ samples out of that $n+1$ samples?

Thanks.

Answer 1

The question asks about how the estimation variance of the parameters in least squares fitting changes when an additional observation is included in the data.

Letting $A$ be the model matrix ($A = (a)$ or $A=(a,b)'$ in the examples in the question), $X$ be the parameter vector ($X = (x)$ in the question), and $Y$ be the response vector ($Y = (y_1)$ or $Y=(y_1,y_2)'$ in the question), and introducing a variance-covariance matrix $\Sigma = \sigma^2\mathbb{I}$ for the responses ($\mathbb{I}$ is an identity matrix), the model can be expressed as

$$\mathbb{E}[Y] = AX;\quad\text{Var}[Y] = \Sigma = \sigma^2\mathbb{I}.$$

The least-squares solution is

$$\widehat{X} = (A'A)^{-1}A'Y$$

from which its variance-covariance matrix can be computed as

$$\text{Var}(\widehat{X}) = (A'A)^{-1}A'\text{Var}(Y Y')\left( (A'A)^{-1}A'\right)'= \sigma^2 (A'A)^{-1}A'\left( A (A'A)^{-1}\right) = \sigma^2 (A'A)^{-1}.$$

For the examples in the question (assuming $a\ne 0$) these values are $\sigma^2 / a^2$ and $\sigma^2 / (a^2 + b^2).$ Obviously, because $a^2+b^2\ge a^2,$ the variance does not increase in this case when the second observation is added.

---

In general, let $\vec{a}$ be a new row to be adjoined to $A$. This changes the precision matrix $A'A$ into $A'A + \vec{a}'\vec{a}$. The original precision matrix is positive-semidefinite, which means that for all vectors $u$

$$u' A'A u \ge 0.$$

Now for an arbitrary vector $u,$

$$u'(A'A + \vec{a}'\vec{a}) u = u'(A'A)u + u'(\vec{a}'\vec{a})u = u'(A'A)u + (\vec{a}u)^2 \ge u'(A'A)u.$$

In this sense $A'A$ becomes "more" positive-definite when another observation is adjoined to it. We might say, in somewhat descriptive language, that the estimate $\widehat{X}$ becomes more precise as new observations are added. Indeed, it becomes strictly more precise whenever any nonzero observation is added. This is merely the vector analog of the earlier argument that $a^2+b^2 \ge a^2.$

A similar analysis can be carried in the generalized least squares setting when arbitrary variance-covariance matrices $\Sigma$ describe the distribution of the $z_i$ (the "error structure"), with the same conclusion: including additional nonzero observations always increases the precision of the parameter estimates.