# Sequential linear estimation: Is MSE non-increasing?

Suppose we estimate $x$ from $y_1=ax+z_1$ and get $x_1$. Then suppose we receive $y_2=bx+z_2$ and reestimate $x$ using both $y_1$ and $y_2$ and get $x_2$. Is the means squared error (MSE) of $x_2$ always lower than MSE of $x_1$. In other words is $$E(x-x_1)^2\geq E(x-x_2)^2?$$ Here $a,b$ are know constants and $z_1$ and $z_2$ are Gaussian errors and the estimation is linear minimum mean squared error. The equations can also be vectors in that case error is the trance of the error covariance matrix. By this wiki it seems to be decreasing.

Edit: explanation

At time $t=0$ a real value $x$ is generated by a random process $X$. Then at time $t=1$, I receive a noise corrupted and scaled version $y_1$ of $x$ denoted by $y_1=ax+z_1$. From $y_1$ I do linear minimum mean squared error (LMMSE) estimation and estimate $x$ and get $x_1$ and the corresponding LMMSE is $E(x-x_1)^2$. Here $a$ is a constant and is known, $z_1$ is Gaussian noise with known mean and variance. LMMSE estimation is straightforward.

Then at time $t=2$, I receive another noise corrupted and scaled version $y_2$ of that same $x$ which is denoted by $y_2=bx+z_2$. Again $b$ is a known constant and $z_2$ is Gaussian noise with known mean and variance. Now I simultaneously use both $y_1$ and $y_2$ to estimate $x$ by LMMSE estimation and I get the value $x_2$ and the corresponding error $E(x-x_2)^2$. My question: is $E(x-x_1)^2\geq E(x-x_2)^2?$. i.e., the time actually does not matter the question is does LMMSE estimation using $n+1$ samples always give a less error than estimating only using $n$ samples out of that $n+1$ samples?

Thanks.

• Could you please explain the kinds of data and the process used to "estimate" a value $x$ from equations of the form $y=ax+z$? – whuber Nov 27 '13 at 19:37
• I am using linear minimum mean squared error estimation. I receive two corrupted copies of the same sample value $x$ and want to compare the estimate of $x$ with respect to one observation against both observations. The process which generated $x$ can be assumed to be Gaussian and we know all covariances matrices. Thanks. – triomphe Nov 27 '13 at 20:22
• Thanks, but what you're doing is still not quite clear. It would seem that in the first case you have triples of data in the form $(y, a, z)$ and you are estimating the parameter $x$ using least squares, but this does not match your question or the description in your comment. Including a small example of a dataset and the estimates within your question might help overcome these difficulties in communicating the context of your problem. – whuber Nov 27 '13 at 20:27
• @whuber I included an explanation. But I don't have data. This is a theoretical question. Is this question not valid? Thanks. – triomphe Nov 27 '13 at 20:42
• I cannot say whether it is valid until it becomes understandable. Your recent edits help somewhat, but you refer to "corrupted and scaled." Presumably this means $a$ and $z_2$ are unknown and somehow are being considered as realizations of random variables. Precisely what do you assume about these variables and specifically how do you use this assumption to estimate $x_1$? – whuber Nov 27 '13 at 21:39

The question asks about how the estimation variance of the parameters in least squares fitting changes when an additional observation is included in the data.

Letting $A$ be the model matrix ($A = (a)$ or $A=(a,b)'$ in the examples in the question), $X$ be the parameter vector ($X = (x)$ in the question), and $Y$ be the response vector ($Y = (y_1)$ or $Y=(y_1,y_2)'$ in the question), and introducing a variance-covariance matrix $\Sigma = \sigma^2\mathbb{I}$ for the responses ($\mathbb{I}$ is an identity matrix), the model can be expressed as

$$\mathbb{E}[Y] = AX;\quad\text{Var}[Y] = \Sigma = \sigma^2\mathbb{I}.$$

The least-squares solution is

$$\widehat{X} = (A'A)^{-1}A'Y$$

from which its variance-covariance matrix can be computed as

$$\text{Var}(\widehat{X}) = (A'A)^{-1}A'\text{Var}(Y Y')\left( (A'A)^{-1}A'\right)'= \sigma^2 (A'A)^{-1}A'\left( A (A'A)^{-1}\right) = \sigma^2 (A'A)^{-1}.$$

For the examples in the question (assuming $a\ne 0$) these values are $\sigma^2 / a^2$ and $\sigma^2 / (a^2 + b^2).$ Obviously, because $a^2+b^2\ge a^2,$ the variance does not increase in this case when the second observation is added.

In general, let $\vec{a}$ be a new row to be adjoined to $A$. This changes the precision matrix $A'A$ into $A'A + \vec{a}'\vec{a}$. The original precision matrix is positive-semidefinite, which means that for all vectors $u$

$$u' A'A u \ge 0.$$

Now for an arbitrary vector $u,$

$$u'(A'A + \vec{a}'\vec{a}) u = u'(A'A)u + u'(\vec{a}'\vec{a})u = u'(A'A)u + (\vec{a}u)^2 \ge u'(A'A)u.$$

In this sense $A'A$ becomes "more" positive-definite when another observation is adjoined to it. We might say, in somewhat descriptive language, that the estimate $\widehat{X}$ becomes more precise as new observations are added. Indeed, it becomes strictly more precise whenever any nonzero observation is added. This is merely the vector analog of the earlier argument that $a^2+b^2 \ge a^2.$

A similar analysis can be carried in the generalized least squares setting when arbitrary variance-covariance matrices $\Sigma$ describe the distribution of the $z_i$ (the "error structure"), with the same conclusion: including additional nonzero observations always increases the precision of the parameter estimates.

• Thanks. I have a few clarifications please. Why are the noise terms $z_1,z_2$ not in the model? And Wikipedia gives the linear minimum means squared error estimate as $\hat{x}=C_XA^T(AC_XA^T+C_Z)^{-1}(y-AE[X])+E[X]$, where $C_X$, $C_Z$ is the covariance of X and covariance of $Z$ and $E$ is expectation. I can get to you answer by assuming many $C_X=1$, $C_Z=0$ and $E[X]=0$. Is your answer still valid for the general case? – triomphe Nov 29 '13 at 23:05
• And next question is why do you consider $Var(X)$ and not the error variance $Var(X-\hat X)$? Thanks. en.wikipedia.org/wiki/… – triomphe Nov 29 '13 at 23:06
• The "noise terms" are in the model: I have stipulated their means ($0$) and covariances ($\Sigma$). The rest of your comment is inconsistent with your question: the "covariance of $X$ and ... $Z$" makes no sense in this context because $X$ is a parameter, not a random variable. – whuber Nov 30 '13 at 17:41
• But my model is the same model as shown in the wikipedia article. $X$ is not a parameter it is a random variable. I am trying to estimate the current sample value of $X$. I am doing linear minimum mean squared estimation, not least square, and I know they have lot in common but in LMMSE $X$ is always a random variable, right? Could you please explain what changes in your answer then? I am not very familiar with stat so may be my terms are confusing. Thanks a lot. – triomphe Nov 30 '13 at 18:37