I'm trying to learn a regression model for a computer vision / pattern recognition task, where I try to estimate a continuous variable from a set of visual features.

I have done preliminary experiments and saw that some features perform better compared to others, I measure the performance with MAE (mean absolute error), so I have such a table:

Feature   MAE
-------  -----
F1        5.5
F2        7.5
F3        8.5
etc.


I see that F1 performs the best, i.e. yields the least cross validation error. The other features are not so bad either, but simple feature-level or score-level fusion does not decrease the MAE (it remains around 5.6 or so), so with this level of knowledge, I can't make use of F2 and F3. So currently my best model is the one that just uses F1. But I want to -if possible- include the information from other features as well. My learning strategy is instance level normalization + ridge regression. All the features have the same dimensionality (1000), and I have a dataset of 4000 samples.

So my question is, are there alternative approaches that are commonly used for this type of situation? The alternatives I can think of are:

1. Weighted score-level fusion (however I feel it is prone to overfitting, and it does not improve the accuracy that much)
2. Dimension reduction (of F2 and F3) with PCA. (This didn't help either. Also, I feel like there is no point of choosing the maximally-variant subspace other than reducing the risk of overfitting because of the reduced dimensionality)
3. Dimension reduction (of F2 and F3) with CCA (canonical correlation analysis) to select a subspace that is more "relevant" to the labels. I didn't actually try this because I don't know how to, but will it be beneficial? Also can CCA be applied to regression? I previously applied it to C-class classification to obtain a C-1 dimensional subspace, but does it make sense for regression?

And are there other alternatives that can enable the use of the "bad" features such as F2 and F3?

Thanks for any help,

You could consider using feature wrappers (e.g. in R caret) to find the optimal combination of features. In contrast to feature filters (which consider the inter-feature-correlation and feature-target correlation), wrappers also include the model in their evaluation, therefore measure the model performance. They usually either incrementally add or remove features from the currently used feature set, until a reasonable combination of features is found that optimized your model performance.

If your data shows correlation between features (which might be the case if it represents e.g. visual information in some form of spatial way), then dimensionality reduction could in fact make sense, simply due to visual sensing likely only observing a subspace of what could be represented in the whole feature space. Besides using PCA, you could consider using Partial least squares (PLS) regression or similar approaches for this purpose (you could think of PLS regression as a supervised sibling to PCA, as it additionally considers the target variable).

Further, it might be the case that the feature-target-relation is too complex for your current model, in which case you would simply have a bias problem and get stuck with a certain amount of error even in training. Using a more powerful model could help in this case.