How to combine 2 different observations to improve state estimates? Context
Let $\mathbf{x}_i \in \mathbb{R}^{100}$ and $\mathbf{z}_i \in \mathbb{R}^{20}$ be input vectors with the same corresponding target $\mathbf{y}_i \in \mathbb{R}^{25}$.
Using ridge regression we obtain two separate mappings, $\mathbf{A}_\mathbf{x}$ and $\mathbf{A}_\mathbf{z}$, such that
$$\mathbf{A}_\mathbf{x}\mathbf{x}_i = \mathbf{y}_i + \mathbf{\epsilon}_\mathbf{x},$$
$$\mathbf{A}_\mathbf{z}\mathbf{z}_i = \mathbf{y}_i + \mathbf{\epsilon}_\mathbf{z},$$
with $\mathbf{\epsilon}$ denoting noise.
I wish to minimise the noise (error) by considering both observations $\mathbf{x}_i$ and $\mathbf{z}_i$.
Questions
How can I combine the observations ($\mathbf{x}$ and $\mathbf{z}$) to obtain a lower error on average?
I have two approaches, neither of which give satisfactory results on average:

*

*Append the two vectors [$\mathbf{x}$ ; $\mathbf{z}$] and learn a new mapping.


*Use a weighted average of the two estimates.
For (1), do I need to do any preprocessing before appending the vectors, such as spatial alignment (PCA?) or some form of value normalisation?
For (2), is there a way to automatically determine a suitable weighting?
What other ways are likely to yield better results?
How can I analyse the complementary nature of the two observations in relation to the state being estimated?
 A: First of all, the training error obtained by solution #1 (concatenating $x$ and $z$) can't be higher than the error obtained by regression against $x$ or $z$ alone. But of course, the test error (i.e., prediction performance on new data) may well be worse.  
If I were confronted with this problem, I would normalize $x$ and $z$ to have the same marginal variance, then concatenate them and then attempt to vary the ridge parameter $\lambda$ -- I would bet that for some setting of $\lambda$ you can find performance that's better than you get from regressing against $x$ or $z$ separately.  
However, if $x$ and $z$ are highly correlated, then you might wish to try Principal Components Regression instead.  The basic idea is to compute the principal components of $[x;z]$, project onto the subspace spanned by the first $n$ principal components, then perform standard regression using the projected data as your regressors. You can now vary $n$ and the ridge parameter (if you like) to explore the space of possible solutions using cross-validation.
You can of course also try other regularization methods: Lasso (L1 regularization) is more appropriate if you expect your regression coefficients to be sparse (several large values and many zeros); "elastic net" trades off Ridge and Lasso (L1 and L2) forms of regularization.
