I'm trying to learn a regression model for a computer vision / pattern recognition task, where I try to estimate a continuous variable from a set of visual features.

I have done preliminary experiments and saw that some features perform better compared to others, I measure the performance with MAE (mean absolute error), so I have such a table:

Feature   MAE
-------  -----
F1        5.5
F2        7.5
F3        8.5
etc.


I see that F1 performs the best, i.e. yields the least cross validation error. The other features are not so bad either, but simple feature-level or score-level fusion does not decrease the MAE (it remains around 5.6 or so), so with this level of knowledge, I can't make use of F2 and F3. So currently my best model is the one that just uses F1. But I want to -if possible- include the information from other features as well. My learning strategy is instance level normalization + ridge regression. All the features have the same dimensionality (1000), and I have a dataset of 4000 samples.

So my question is, are there alternative approaches that are commonly used for this type of situation? The alternatives I can think of are:

1. Weighted score-level fusion (however I feel it is prone to overfitting, and it does not improve the accuracy that much)
2. Dimension reduction (of F2 and F3) with PCA. (This didn't help either. Also, I feel like there is no point of choosing the maximally-variant subspace other than reducing the risk of overfitting because of the reduced dimensionality)
3. Dimension reduction (of F2 and F3) with CCA (canonical correlation analysis) to select a subspace that is more "relevant" to the labels. I didn't actually try this because I don't know how to, but will it be beneficial? Also can CCA be applied to regression? I previously applied it to C-class classification to obtain a C-1 dimensional subspace, but does it make sense for regression?

And are there other alternatives that can enable the use of the "bad" features such as F2 and F3?

Thanks for any help,