I have a set of features extracted from the same samples and I'm learning a kernel ridge regression. Now, especially for feature fusion, reducing the number of features before combining them seems like a good idea. However, I have some concerns about this:

  1. PCA will not necessarily give me the most relevant features
  2. Since I'm learning the regression function from the kernel, its dimensionality will not change after dimension reduction with PCA.

(Details of the dataset: I have $\sim7000$ samples (which are videos) and the features are visual descriptors like LBP-TOP, LPQ-TOP etc., which usually have $>5000$ dimensions.)

When I try to do it, I see that each feature has its k-fold cross validation error increased a little. So is it actually a good idea to reduce the original features? If so, are there better options than PCA?

I also suspect if the kernel is too big, especially for the reduced data. I think having more samples is usually a good thing, but for the case of kernel regression, can it actually be increasing the complexity, hence overlearning occurs? I'm optimizing the regularization coefficient, not sure if it completely prevents overfitting though. So does it make sense to somehow reduce the kernel itself?

Thanks for any help,

  • $\begingroup$ You mention the kernel size, so it sounds like you're using an RBF kernel. Are you optimizing this hyperparameter? The optimal value may be different for PCA-transformed vs. untransformed data. I don't understand the meaning of your point (2). $\endgroup$ – user20160 Jun 11 '16 at 22:35
  • $\begingroup$ I'm using a linear kernel, just the multiplication of each data point. If you mean the last paragraph, since we are learning the regression from the inverse of a bigger matrix (kernel), I wonder if sample size is negatively affecting the generalization performance. $\endgroup$ – jeff Jun 12 '16 at 0:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.