0
$\begingroup$

I am trying to create a classifier using SVM on a dataset that is composed of 6 sets of data for each of my observations. When I train the SVM (rbf kernel), I get a better performance of the classifier when I use only 2 out of 6 sets of features (reducing the number of features to about half): when I add features, my accuracy degrades by over 5%. I understand that "junk in, junk out", but we were surprised by the magnitude of this effect. Note that I scale the features to all have the same range and variance before training the SVM (using sklearn's MinMaxScaler()).

Our interpretation of this result is that we are not representing the feature space correctly - by calculating a single kernel on all 6 sets of features at the same time, I lose information from each single set of features. To make up for this, I want to calculate a kernel on each feature set independently (and so also find the optimal gamma separately) and then concatenate the results to train the classifier (a bit like what described here). Basically, I would like to do:

$$ k_{final} = k_1(X_1) + k_2(X_2) + k_3(X_3) + k_4(X_4) + k_5(X_5) + k_6(X_6) $$

  1. Is this something that can be done and a sensible thing to do? And would it help with my problem?
  2. If so, can I implement this in sklearn? If so, how?

I have found a way to calculate kernels in sklearn (https://scikit-learn.org/stable/modules/gaussian_process.html#gp-kernels), and so I could calculate the kernel on each set separately. However, once I output the distance matrix, I am not sure how I would use it to train the SVM.

My understanding of kernels is only very intuitive, so I would appreciate an answer with limited maths!

Improve this question
$\endgroup$
3
  • $\begingroup$ The sum of kernels is also a kernel, so from that perspective it's valid. stats.stackexchange.com/questions/270967/… // Whether or not it's a good idea for you problem is, of course problem-dependent, so I'm not sure we can answer that directly. // "How do I ___ in [python/R/Excel]" is not an on-topic question here. $\endgroup$
    – Sycorax ♦
    Commented May 29, 2020 at 15:45
  • $\begingroup$ @SycoraxsaysReinstateMonica thank you for pointing me to that question! One more question: if I am using different kernels, do I still need to scale the features beforehand or does that become less important (assuming they are all similar within one subset)? $\endgroup$
    – mm523
    Commented May 29, 2020 at 15:56
  • $\begingroup$ Generally, scaling gives better results, but this isn't always true. See: stats.stackexchange.com/questions/172795/… $\endgroup$
    – Sycorax ♦
    Commented May 29, 2020 at 16:00

0

Reset to default