I am trying to verify a certain derivation of linear minimum mean square error estimator as it appears in [1] (also called linear a posteriori estimate, since the estimate is based on measurement).
Linear estimator is assumed $\hat{\mathbf{x}}(\mathbf{z}) = \mathbf{Az} + \mathbf{b}$, where $\mathbf{A} \in \mathbb{R}^{n\times p},\ \mathbf{b} \in \mathbb{R}^{n}$ are constant and $\mathbf{z} \in \mathbb{R}^{p}$ is measurement of $\mathbf{x} \in \mathbb{R}^n$ (both $\mathbf{x}$ and $\mathbf{z}$ are random variables).
The goal is to find such $\mathbf{A},\,\mathbf{b}$ that minimize $J = \overline{(\mathbf{x} - \hat{\mathbf{x}})^T(\mathbf{x} - \hat{\mathbf{x}})}$. (Overbar denotes expectation.) The difficulty I am facing is in simplifying this expression to the form presented in the book.
The derivation in [1] proceeds as follows: $$ J = \overline{(\mathbf{x} - \hat{\mathbf{x}})^T(\mathbf{x} - \hat{\mathbf{x}})} = \mathrm{Tr}\ \overline{(\mathbf{x} - \hat{\mathbf{x}})^T(\mathbf{x} - \hat{\mathbf{x}})} = \mathrm{Tr}\ \overline{(\mathbf{x} - \mathbf{Az} - \mathbf{b})(\mathbf{x} - \mathbf{Az} - \mathbf{b})^T} = \mathrm{Tr}\ \overline{\left[(\mathbf{x} - \bar{\mathbf{x}}) - (\mathbf{Az} + \mathbf{b} - \bar{\mathbf{x}})\right]\left[(\mathbf{x} - \bar{\mathbf{x}}) - (\mathbf{Az} + \mathbf{b} - \bar{\mathbf{x}})\right]^T} $$ I the last equality the ''smart zero'' $ -\bar{\mathbf{x}} + \bar{\mathbf{x}}$ was added. Here I am still able to follow. Authors then write (and that's where the problem lies) ''After some straighforward work we arrive at'' $$ J = \mathrm{Tr}\ \left[\mathbf{P}_x + \mathbf{A}(\mathbf{P}_z + \overline{\mathbf{zz^T}})\mathbf{A}^T + (\mathbf{b} - \bar{\mathbf{x}})(\mathbf{b} - \bar{\mathbf{x}})^T + 2\mathbf{A}\bar{\mathbf{z}}(\mathbf{b} - \bar{\mathbf{x}})^T - 2\mathbf{AP}_{zx} \right] $$ where $$\mathbf{P}_x = \overline{(\mathbf{x} - \bar{\mathbf{x}})(\mathbf{x} - \bar{\mathbf{x}})^T}, \ \mathbf{P}_{zx} = \overline{(\mathbf{z} - \bar{\mathbf{z}})(\mathbf{x} - \bar{\mathbf{x}})^T} $$ (analogously for $\mathbf{P}_z$ and $\mathbf{P}_{xz} = \mathbf{P}^T_{zx}$) are covariance matrices.
My work so far
If I multiply out the inner expression under overbar in
$$\mathrm{Tr}\ \overline{\left[(\mathbf{x} - \bar{\mathbf{x}}) - (\mathbf{Az} + (\mathbf{b} - \bar{\mathbf{x}}))\right]\left[(\mathbf{x} - \bar{\mathbf{x}}) - (\mathbf{Az} + (\mathbf{b} - \bar{\mathbf{x}}))\right]^T}
$$
I get
$$
\mathrm{Tr}\ \left[\overline{(\mathbf{x} - \mathbf{\bar{x}})(\mathbf{x} - \mathbf{\bar{x}})^T} - 2\overline{(\mathbf{x} - \mathbf{\bar{x}})(\mathbf{Az} + (\mathbf{b} - \bar{\mathbf{x}}))^T} + \overline{(\mathbf{Az} + (\mathbf{b} - \bar{\mathbf{x}}))(\mathbf{Az} + (\mathbf{b} - \bar{\mathbf{x}}))^T}\right]
$$
Working out individual terms:
The first term: $\overline{(\mathbf{x} - \bar{\mathbf{x}})(\mathbf{x} - \bar{\mathbf{x}})^T} = \mathbf{P}_x$
The second term: (utilizing ''smart zero'' $-\mathbf{A}\bar{\mathbf{z}} + \mathbf{A}\bar{\mathbf{z}}$ and the fact that $\overline{(\mathbf{x} - \bar{\mathbf{x}})\bar{\mathbf{z}}\mathbf{A}^T} = 0, \ \overline{(\mathbf{x} - \bar{\mathbf{x}})(\mathbf{b} - \bar{\mathbf{x}})^T} = 0$)
$-2\overline{(\mathbf{x} - \mathbf{\bar{x}})(\mathbf{Az} + (\mathbf{b} - \bar{\mathbf{x}}))^T} = -2\overline{(\mathbf{x} - \mathbf{\bar{x}})(\mathbf{A}(\mathbf{z}-\bar{\mathbf{z}}) + \mathbf{A}\mathbf{\bar{z}} + (\mathbf{b} - \bar{\mathbf{x}}))^T} = -2\mathbf{AP}_{zx}$The third term: (''smart zero'' $-\mathbf{A}\bar{\mathbf{z}} + \mathbf{A}\bar{\mathbf{z}}$ again! <-- this hit me when writing this post, *sigh*, fuuuu)
$\overline{(\mathbf{Az} + (\mathbf{b} - \bar{\mathbf{x}}))(\mathbf{Az} + (\mathbf{b} - \bar{\mathbf{x}}))^T} = \mathbf{A}\bar{\mathbf{z}}\bar{\mathbf{z}}^T\mathbf{A}^T + \mathbf{A}\mathbf{P}_z\mathbf{A}^T + 2\mathbf{A}\bar{\mathbf{z}}(\mathbf{b}-\bar{\mathbf{x}})^T + (\mathbf{b} - \bar{\mathbf{x}})(\mathbf{b} - \bar{\mathbf{x}})^T$
This way I am able to account for all the terms in the objective function $J$ and, of course, everything is solved.
Although, I still have one question:
Authors in [1] have $\mathbf{A}(\mathbf{P}_z + \overline{\mathbf{zz}}^T)\mathbf{A}^T$ whereas what I obtained is $\mathbf{A}(\mathbf{P}_z + \mathbf{\bar{z}}\mathbf{\bar{z}}^T)\mathbf{A}^T$. But I think that $\overline{\mathbf{zz}}^T \neq \mathbf{\bar{z}}\mathbf{\bar{z}}^T$ and therefore, unless authors of [1] used some other devilish approach, there's a typo in the book. What do you think?
[1]: F.L. Lewis, L. Xie, D. Popa, Optimal and Robust Estimation: With an Introduction to Stochastic Control Theory, 2008
