I have a set of features extracted from the same samples and I'm learning a kernel ridge regression. Now, especially for feature fusion, reducing the number of features before combining them seems like a good idea. However, I have some concerns about this:
- PCA will not necessarily give me the most relevant features
- Since I'm learning the regression function from the kernel, its dimensionality will not change after dimension reduction with PCA.
(Details of the dataset: I have $\sim7000$ samples (which are videos) and the features are visual descriptors like LBP-TOP, LPQ-TOP etc., which usually have $>5000$ dimensions.)
When I try to do it, I see that each feature has its k-fold cross validation error increased a little. So is it actually a good idea to reduce the original features? If so, are there better options than PCA?
I also suspect if the kernel is too big, especially for the reduced data. I think having more samples is usually a good thing, but for the case of kernel regression, can it actually be increasing the complexity, hence overlearning occurs? I'm optimizing the regularization coefficient, not sure if it completely prevents overfitting though. So does it make sense to somehow reduce the kernel itself?
Thanks for any help,