# Kernelization vs pre-defined basis functions: which one is better and why?

I am learning about kernels and how linear models can use them to model nonlinear data. Consider, for example, linear regression for nonlinear function $$y(\textbf{x})$$. The idea is to project the feature (data) vectors $$\textbf{x}_i$$ to a higher dimensional space via a mapping $$\phi(\textbf{x}_i)$$, where the function $$y(\phi(\textbf{x}))$$ becomes linear. Then it turns out that the regression problem only depends on the dot products $$\phi(\textbf{x}_i)^T \phi(\textbf{x}_j)$$ so we can introduce a kernel $$k(\textbf{x}_i, \textbf{x}_j) = \phi(\textbf{x}_i)^T \phi(\textbf{x}_j)$$ and we can write our predictor as $$\hat{y}(\textbf{x}) = \sum_i^n a_i k(\textbf{x}_i, \textbf{x})$$ where $$n$$ is the number of training points and $$a_i$$ minimize the cost function.

The properties of $$k(\textbf{x}_i, \textbf{x}_j)$$ are particular because it must represent a dot product.

Now my question is, how is this kernel formalism different and why is it better than introducing a set of basis functions to begin with? I could say that I define $$m \ne n$$ basis functions $$\varphi_l(\textbf{x})$$ and write my estimator in this basis: $$\hat{y}(\textbf{x}) = \sum_l^m b_l \varphi_l(\textbf{x})$$. Then I minimize the cost w.r.t $$b_l$$. Because I choose the basis $$\varphi_l(\textbf{x})$$, there are now no restrictions on its form because it does not have to represent any dot product. Also, I can choose $$m \ll n$$ so the number of evaluations can be smaller than in the kernel case.

I assume there are good reasons to use the kernel formalism (i.e. projecting to high dimensional space and define a corresponding kernel) vs. the function expansion and I would be curious what those reasons are.

• the kernel perspective is useful when there are significantly fewer datapoints than there are members of the basis. For example, in infinite dimensional space. Feb 21, 2023 at 13:50