# What are the primary advantages of using Kernels in predicting continuous outcomes?

Consider the linear model: $$y = X \theta + \epsilon$$ with $X$ inputs or features of inputs and $\theta$ a vector of parameters (and $\epsilon$ the error) with regularized error function $$J(w)= \frac{1}{2} [X \theta - y]^T[X \theta - y] + \frac{\lambda}{2}\theta^T\theta.$$

The idea of kernels $k(x_i,x_j)$ is to define a similarity function between all observation pairs $i,j$ and summarize them in the Gram Matrix $K$. If we define $K=XX^T$, then $J(w)$ can be re-written as $$J(a)= \frac{1}{2}a^TKKa-a^TKt+\frac{1}{2}t^Tt+\frac{\lambda}{2}a^TKa$$ where $$a = (K + \lambda I_N)^{-1} t.$$

Finally a prediction for $y$ can be made as $$\hat{y}(x)=k(x)^T(K+\lambda I_N)^{-1}t.$$ I understand that, while it is amazing that we can eliminate the parameters completely from the prediction equation, the computational burden strongly increases because $K$ needs to be inverted and is of order $N \times N$.

What I do not understand is: what are the advantages of using a Kernel in this model?

• On a uniform domain, for non-sick domain sizes, the inverse is relatively cheap. Also, it only has to be computed once. Oct 10, 2016 at 21:38
• This regression will provide a smooth (and exact, if there's no error) interpolation between the observed data points; how and how sharply the function changes between the data points is informed by the choice of kernel and kernel hyper-parameters.
– Sycorax
Oct 10, 2016 at 21:39
• @Sycorax So the idea is to find a better - nonlinear - representation of the data by smartly choosing Kernels? And the choice is perhaps guided by cross-validation? And that's it? Oct 10, 2016 at 21:41
• Basically. You could cross-validate. On the other hand, you could explicitly model hyperparameters, with MCMC or a variant to average out the effect of alternative hyperparameters. Usually people assume that the RBF kernel is "good enough" and get on with their lives (cf "no one ever got fired for running OLS regression"), but there are a rich diversity of options. There's a good discussion in Gaussian Processes for Machine Learning and also Gelman's BDA3.
– Sycorax
Oct 10, 2016 at 21:46

The following is from Christopher M. Bishop's "Pattern Recognition and Machine Learning" page 294 where $M$ represents the feature dimension:
"In the dual formulation, we determine the parameter vector a by inverting an $N ×N$ matrix, whereas in the original parameter space formulation we had to invert an $M × M$ matrix in order to determine w. Because N is typically much larger than M, the dual formulation does not seem to be particularly useful. However, theadvantage of the dual formulation, as we shall see, is that it is expressed entirely interms of the kernel function $k(x, x)$. We can therefore work directly in terms of kernels and avoid the explicit introduction of the feature vector $\phi(x)$, which allows us implicitly to use feature spaces of high, even infinite, dimensionality."