I am learning about kernels and how linear models can use them to model nonlinear data. Consider, for example, linear regression for nonlinear function $y(\textbf{x})$. The idea is to project the feature (data) vectors $\textbf{x}_i$ to a higher dimensional space via a mapping $\phi(\textbf{x}_i)$, where the function $y(\phi(\textbf{x}))$ becomes linear. Then it turns out that the regression problem only depends on the dot products $\phi(\textbf{x}_i)^T \phi(\textbf{x}_j)$ so we can introduce a kernel $k(\textbf{x}_i, \textbf{x}_j) = \phi(\textbf{x}_i)^T \phi(\textbf{x}_j)$ and we can write our predictor as $\hat{y}(\textbf{x}) = \sum_i^n a_i k(\textbf{x}_i, \textbf{x})$ where $n$ is the number of training points and $a_i$ minimize the cost function.

The properties of $k(\textbf{x}_i, \textbf{x}_j)$ are particular because it must represent a dot product.

Now my question is, how is this kernel formalism different and why is it better than introducing a set of basis functions to begin with? I could say that I define $m \ne n$ basis functions $\varphi_l(\textbf{x})$ and write my estimator in this basis: $\hat{y}(\textbf{x}) = \sum_l^m b_l \varphi_l(\textbf{x})$. Then I minimize the cost w.r.t $b_l$. Because I choose the basis $\varphi_l(\textbf{x})$, there are now no restrictions on its form because it does not have to represent any dot product. Also, I can choose $m \ll n$ so the number of evaluations can be smaller than in the kernel case.

I assume there are good reasons to use the kernel formalism (i.e. projecting to high dimensional space and define a corresponding kernel) vs. the function expansion and I would be curious what those reasons are.

  • $\begingroup$ the kernel perspective is useful when there are significantly fewer datapoints than there are members of the basis. For example, in infinite dimensional space. $\endgroup$ Feb 21, 2023 at 13:50


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.