Let $\mathbf{x}_i \in \mathbb{R}^{100}$ and $\mathbf{z}_i \in \mathbb{R}^{20}$ be input vectors with the same corresponding target $\mathbf{y}_i \in \mathbb{R}^{25}$.

Using ridge regression we obtain two separate mappings, $\mathbf{A}_\mathbf{x}$ and $\mathbf{A}_\mathbf{z}$, such that

$$\mathbf{A}_\mathbf{x}\mathbf{x}_i = \mathbf{y}_i + \mathbf{\epsilon}_\mathbf{x},$$

$$\mathbf{A}_\mathbf{z}\mathbf{z}_i = \mathbf{y}_i + \mathbf{\epsilon}_\mathbf{z},$$

with $\mathbf{\epsilon}$ denoting noise.

I wish to minimise the noise (error) by considering both observations $\mathbf{x}_i$ and $\mathbf{z}_i$.


How can I combine the observations ($\mathbf{x}$ and $\mathbf{z}$) to obtain a lower error on average?

I have two approaches, neither of which give satisfactory results on average:

  1. Append the two vectors [$\mathbf{x}$ ; $\mathbf{z}$] and learn a new mapping.

  2. Use a weighted average of the two estimates.

For (1), do I need to do any preprocessing before appending the vectors, such as spatial alignment (PCA?) or some form of value normalisation?

For (2), is there a way to automatically determine a suitable weighting?

What other ways are likely to yield better results?

How can I analyse the complementary nature of the two observations in relation to the state being estimated?


First of all, the training error obtained by solution #1 (concatenating $x$ and $z$) can't be higher than the error obtained by regression against $x$ or $z$ alone. But of course, the test error (i.e., prediction performance on new data) may well be worse.

If I were confronted with this problem, I would normalize $x$ and $z$ to have the same marginal variance, then concatenate them and then attempt to vary the ridge parameter $\lambda$ -- I would bet that for some setting of $\lambda$ you can find performance that's better than you get from regressing against $x$ or $z$ separately.

However, if $x$ and $z$ are highly correlated, then you might wish to try Principal Components Regression instead. The basic idea is to compute the principal components of $[x;z]$, project onto the subspace spanned by the first $n$ principal components, then perform standard regression using the projected data as your regressors. You can now vary $n$ and the ridge parameter (if you like) to explore the space of possible solutions using cross-validation.

You can of course also try other regularization methods: Lasso (L1 regularization) is more appropriate if you expect your regression coefficients to be sparse (several large values and many zeros); "elastic net" trades off Ridge and Lasso (L1 and L2) forms of regularization.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.