I want to perform multiple kernel learning on my dataset and apply each (rbf) kernel to a different subset of features to then combine them. I do not want to have the same kernel with a range of parameters (as suggested here) but rather find the optimal parameters for each component kernel first before combining them, and then optimise the weights (as suggested here).

I am wondering what metric should I be using to optimise each kernel? Would maximising the margin between the negative and positive class be appropriate? Or minimising the ratio between the radius of Minimum Enclosing Ball (MEB) and the margin?

UPDATE: I have come up with my own solution (accepted answer), but any feedback/comments would be highly appreciated.


I am going to post my own solution to the problem, any feedback on this will be appreciated!

I used as metric of similarity the alignment between the kernel and the ideal kernel (in which I have a value of 1 for samples with the same label and of -1 for samples with different labels). This is a number between 0 and 1, the higher the better.

To find the optimal value of gamma for the rbf kernel, I apply nested CV:

  1. I take out a validation fold (20%) which will not be used to find the optimal $\gamma$
  2. Within the train folds, I create an internal K-fold split in the dataset 3. In the internal train fold, I search over a range of values of $\gamma$ and see which one gives me the best alignment with the ideal kernel for the internal train set $$ score_{train}(\gamma) = alignment(K_{internal-train}, K_{ideal-internal-train}) $$ 4. I take the $\gamma$ for which $score_{train}(\gamma)$ is maximal and evaluate the performance on the internal test set: $$ score_{test}(\gamma) = alignment(K_{internal-test}, K_{ideal-internal-test}) $$ 5. Repeat steps 2-4 K-fold
  3. Pick the value of $\gamma$ for which $score_{test}(\gamma)$ is max and use it to compute the kernel for the dataset for the MKL and validate on the external validation set.

I repeat this procedure 5-fold to get a performance estimate of my classifier.


It turns out that finding the ideal kernel might overfit the data + the kernel that is the closest to the ideal kernel might not have any information to add when combined with other kernels - i.e. kernels that are weaker on their own can be much more useful when combined.

Instead of using an rbf kernel, I approximated it by using a number of homogeneous polynomial kernels (see this paper), which I combined without selecting. Therefore, for each of my dataset I had D polynomial kernels with grades 1 to D and this seems to yield really good results with my data.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.