1
$\begingroup$

I am trying to verify a certain derivation of linear minimum mean square error estimator as it appears in [1] (also called linear a posteriori estimate, since the estimate is based on measurement).

Linear estimator is assumed $\hat{\mathbf{x}}(\mathbf{z}) = \mathbf{Az} + \mathbf{b}$, where $\mathbf{A} \in \mathbb{R}^{n\times p},\ \mathbf{b} \in \mathbb{R}^{n}$ are constant and $\mathbf{z} \in \mathbb{R}^{p}$ is measurement of $\mathbf{x} \in \mathbb{R}^n$ (both $\mathbf{x}$ and $\mathbf{z}$ are random variables).

The goal is to find such $\mathbf{A},\,\mathbf{b}$ that minimize $J = \overline{(\mathbf{x} - \hat{\mathbf{x}})^T(\mathbf{x} - \hat{\mathbf{x}})}$. (Overbar denotes expectation.) The difficulty I am facing is in simplifying this expression to the form presented in the book.

The derivation in [1] proceeds as follows: $$ J = \overline{(\mathbf{x} - \hat{\mathbf{x}})^T(\mathbf{x} - \hat{\mathbf{x}})} = \mathrm{Tr}\ \overline{(\mathbf{x} - \hat{\mathbf{x}})^T(\mathbf{x} - \hat{\mathbf{x}})} = \mathrm{Tr}\ \overline{(\mathbf{x} - \mathbf{Az} - \mathbf{b})(\mathbf{x} - \mathbf{Az} - \mathbf{b})^T} = \mathrm{Tr}\ \overline{\left[(\mathbf{x} - \bar{\mathbf{x}}) - (\mathbf{Az} + \mathbf{b} - \bar{\mathbf{x}})\right]\left[(\mathbf{x} - \bar{\mathbf{x}}) - (\mathbf{Az} + \mathbf{b} - \bar{\mathbf{x}})\right]^T} $$ I the last equality the ''smart zero'' $ -\bar{\mathbf{x}} + \bar{\mathbf{x}}$ was added. Here I am still able to follow. Authors then write (and that's where the problem lies) ''After some straighforward work we arrive at'' $$ J = \mathrm{Tr}\ \left[\mathbf{P}_x + \mathbf{A}(\mathbf{P}_z + \overline{\mathbf{zz^T}})\mathbf{A}^T + (\mathbf{b} - \bar{\mathbf{x}})(\mathbf{b} - \bar{\mathbf{x}})^T + 2\mathbf{A}\bar{\mathbf{z}}(\mathbf{b} - \bar{\mathbf{x}})^T - 2\mathbf{AP}_{zx} \right] $$ where $$\mathbf{P}_x = \overline{(\mathbf{x} - \bar{\mathbf{x}})(\mathbf{x} - \bar{\mathbf{x}})^T}, \ \mathbf{P}_{zx} = \overline{(\mathbf{z} - \bar{\mathbf{z}})(\mathbf{x} - \bar{\mathbf{x}})^T} $$ (analogously for $\mathbf{P}_z$ and $\mathbf{P}_{xz} = \mathbf{P}^T_{zx}$) are covariance matrices.


My work so far
If I multiply out the inner expression under overbar in $$\mathrm{Tr}\ \overline{\left[(\mathbf{x} - \bar{\mathbf{x}}) - (\mathbf{Az} + (\mathbf{b} - \bar{\mathbf{x}}))\right]\left[(\mathbf{x} - \bar{\mathbf{x}}) - (\mathbf{Az} + (\mathbf{b} - \bar{\mathbf{x}}))\right]^T} $$
I get $$ \mathrm{Tr}\ \left[\overline{(\mathbf{x} - \mathbf{\bar{x}})(\mathbf{x} - \mathbf{\bar{x}})^T} - 2\overline{(\mathbf{x} - \mathbf{\bar{x}})(\mathbf{Az} + (\mathbf{b} - \bar{\mathbf{x}}))^T} + \overline{(\mathbf{Az} + (\mathbf{b} - \bar{\mathbf{x}}))(\mathbf{Az} + (\mathbf{b} - \bar{\mathbf{x}}))^T}\right] $$ Working out individual terms:

  1. The first term: $\overline{(\mathbf{x} - \bar{\mathbf{x}})(\mathbf{x} - \bar{\mathbf{x}})^T} = \mathbf{P}_x$

  2. The second term: (utilizing ''smart zero'' $-\mathbf{A}\bar{\mathbf{z}} + \mathbf{A}\bar{\mathbf{z}}$ and the fact that $\overline{(\mathbf{x} - \bar{\mathbf{x}})\bar{\mathbf{z}}\mathbf{A}^T} = 0, \ \overline{(\mathbf{x} - \bar{\mathbf{x}})(\mathbf{b} - \bar{\mathbf{x}})^T} = 0$)
    $-2\overline{(\mathbf{x} - \mathbf{\bar{x}})(\mathbf{Az} + (\mathbf{b} - \bar{\mathbf{x}}))^T} = -2\overline{(\mathbf{x} - \mathbf{\bar{x}})(\mathbf{A}(\mathbf{z}-\bar{\mathbf{z}}) + \mathbf{A}\mathbf{\bar{z}} + (\mathbf{b} - \bar{\mathbf{x}}))^T} = -2\mathbf{AP}_{zx}$

  3. The third term: (''smart zero'' $-\mathbf{A}\bar{\mathbf{z}} + \mathbf{A}\bar{\mathbf{z}}$ again! <-- this hit me when writing this post, *sigh*, fuuuu)
    $\overline{(\mathbf{Az} + (\mathbf{b} - \bar{\mathbf{x}}))(\mathbf{Az} + (\mathbf{b} - \bar{\mathbf{x}}))^T} = \mathbf{A}\bar{\mathbf{z}}\bar{\mathbf{z}}^T\mathbf{A}^T + \mathbf{A}\mathbf{P}_z\mathbf{A}^T + 2\mathbf{A}\bar{\mathbf{z}}(\mathbf{b}-\bar{\mathbf{x}})^T + (\mathbf{b} - \bar{\mathbf{x}})(\mathbf{b} - \bar{\mathbf{x}})^T$

This way I am able to account for all the terms in the objective function $J$ and, of course, everything is solved.

Although, I still have one question:
Authors in [1] have $\mathbf{A}(\mathbf{P}_z + \overline{\mathbf{zz}}^T)\mathbf{A}^T$ whereas what I obtained is $\mathbf{A}(\mathbf{P}_z + \mathbf{\bar{z}}\mathbf{\bar{z}}^T)\mathbf{A}^T$. But I think that $\overline{\mathbf{zz}}^T \neq \mathbf{\bar{z}}\mathbf{\bar{z}}^T$ and therefore, unless authors of [1] used some other devilish approach, there's a typo in the book. What do you think?


[1]: F.L. Lewis, L. Xie, D. Popa, Optimal and Robust Estimation: With an Introduction to Stochastic Control Theory, 2008

$\endgroup$
1
  • $\begingroup$ To obtain the estimator one needs to take the partial derivatives $$\frac{\partial J}{\partial \mathbf{A}} = 0, \qquad \frac{\partial J}{\partial \mathbf{b}} = 0$$. The resulting linear estimator has the form $$\mathbf{\hat{x}}(\mathbf{z}) = \bar{\mathbf{x}} + \mathbf{P}_{xz}\mathbf{P}^{-1}_z(\mathbf{z} - \bar{\mathbf{z}}) $$ $\endgroup$
    – Jacob
    Jul 24, 2013 at 16:15

1 Answer 1

0
$\begingroup$

Regarding the question about the typo in the book, I found out that it is indeed a typo.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.