# How to combine 2 different observations to improve state estimates?

### Context

Let $$\mathbf{x}_i \in \mathbb{R}^{100}$$ and $$\mathbf{z}_i \in \mathbb{R}^{20}$$ be input vectors with the same corresponding target $$\mathbf{y}_i \in \mathbb{R}^{25}$$.

Using ridge regression we obtain two separate mappings, $$\mathbf{A}_\mathbf{x}$$ and $$\mathbf{A}_\mathbf{z}$$, such that

$$\mathbf{A}_\mathbf{x}\mathbf{x}_i = \mathbf{y}_i + \mathbf{\epsilon}_\mathbf{x},$$

$$\mathbf{A}_\mathbf{z}\mathbf{z}_i = \mathbf{y}_i + \mathbf{\epsilon}_\mathbf{z},$$

with $$\mathbf{\epsilon}$$ denoting noise.

I wish to minimise the noise (error) by considering both observations $$\mathbf{x}_i$$ and $$\mathbf{z}_i$$.

### Questions

How can I combine the observations ($$\mathbf{x}$$ and $$\mathbf{z}$$) to obtain a lower error on average?

I have two approaches, neither of which give satisfactory results on average:

1. Append the two vectors [$$\mathbf{x}$$ ; $$\mathbf{z}$$] and learn a new mapping.

2. Use a weighted average of the two estimates.

For (1), do I need to do any preprocessing before appending the vectors, such as spatial alignment (PCA?) or some form of value normalisation?

For (2), is there a way to automatically determine a suitable weighting?

What other ways are likely to yield better results?

How can I analyse the complementary nature of the two observations in relation to the state being estimated?

First of all, the training error obtained by solution #1 (concatenating $x$ and $z$) can't be higher than the error obtained by regression against $x$ or $z$ alone. But of course, the test error (i.e., prediction performance on new data) may well be worse.
If I were confronted with this problem, I would normalize $x$ and $z$ to have the same marginal variance, then concatenate them and then attempt to vary the ridge parameter $\lambda$ -- I would bet that for some setting of $\lambda$ you can find performance that's better than you get from regressing against $x$ or $z$ separately.
However, if $x$ and $z$ are highly correlated, then you might wish to try Principal Components Regression instead. The basic idea is to compute the principal components of $[x;z]$, project onto the subspace spanned by the first $n$ principal components, then perform standard regression using the projected data as your regressors. You can now vary $n$ and the ridge parameter (if you like) to explore the space of possible solutions using cross-validation.