4
$\begingroup$

In reference to the Representer Theorem in machine learning, Why this is so important? Somehow, this theorem justifies the importance of Kernels in machine learning, i.e. the Kernel trick - a more computationally effective way to "shuttle" data between different dimensions. What does representer theorem in machine learning tells us?

$\endgroup$
0
4
$\begingroup$

The reason the representer theorem is important is that it tells us that the primal model parameters (the weights of the linear model constructed in the kernel induced feature space) can also be written in terms of the dual parameters, where there is one parameter for each training pattern. This means that the output of the linear model can be written as a kernel expansion

$$f(x) = \vec{w}\cdot\phi\{\vec{x}\} + b = \sum_{i=1}^\ell\alpha_i\mathcal{K}(\vec{x},\vec{x}_i) + b$$

Where $\vec{x}_i$ is a training example, $\phi\{\cdot\}$ is the non-linear transformation for which the kernel function $\mathcal{K}:\mathcal{X}\times\mathcal{X}\rightarrow\mathbb{R}$ gives us the inner product, i.e. $\mathcal{K}(\vec{x},\vec{x}') = \left<\phi\{\vec{x}\}\cdot\phi\{\vec{x}\}\right>$. Now the reason this is important is that we know that we can never get a better model by adding any more terms to this expansion, so if we have one term for each training pattern, then we know can obtain the optimal solution given by the primal parameters.

The nice thing about this is that we can construct a model in a potentially infinite dimensional space (such as that induced by the Radial Basis Function kernel) without needing an infinite number of parameters (which we would if we used the primal representation rather than the dual). The representer theorem tells us that this finite optimal representation always exists.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.