Parameter selection in multiple kernel learning

Question

I just got to apply multiple kernel learning to my data recently. I have data from three sources, so I want to learn three RBF kernels for each data source. But the MKL algorithms so far I know assume that the kernel parameters and the cost are fixed. When I used SVM with single RBF kernel before, I need to make a grid search for two variables (gamma and C) and do cross validation as well. I wonder how people in the application field usually deal with the kernel parameters. Take an n+1 dimension exhaustive search for n kernels? Use heuristic? Or make it into a convex problem to solve? Is there any method or tool related? Thanks.

Answer 1

One way to do it is to represent a kernel as a convex combination of kernels having different parameters and have MKL algorithm decide which kernels to use. Here is great paper on MKL algorithms:

Sonnenburg, Sören, et al. "Large scale multiple kernel learning." The Journal of Machine Learning Research 7 (2006): 1531-1565.

Answer 2

The easiest practical approach is to find suitable kernel parameters per data set separately, as you usually would when you only have a single source. When combining the kernels later on, there is usually some form of weighting involved.

The improvement of finding kernel parameters globally (e.g. for each data source based on the accuracy of the final MKL model) is almost always small/insignificant. I therefore recommend finding kernel parameters per single data source and then using a weighting scheme.

Finding kernel parameters globally does not generally lead to a convex optimization problem, and that is the key issue (for example, consider the RBF kernel). Finding weights per kernel is convex (often formulated as a QCLP).