7
$\begingroup$

When a learning algorithm (e.g. classification, regression, clustering or dimension reduction) uses only the dot product between data points $\mathbf {x x^T}$ we can implicitly use a higher dimensional mapping $\phi(\mathbf x)$ through the kernel trick, exchanging every instance where the dot product occurs by the kernel $\mathbf K = \phi(\mathbf x) \phi(\mathbf x) ^ \mathbf T$.

In linear models, SVMs for example, one can account for an intercept adding a constant column to data points. If we use the linear kernel $\mathbf K = \mathbf {x x^T}$ it makes a lot of sense to me to keep that constant column: you can retrieve the column coefficients $\mathbf w$ from the kernel product coefficients $\mathbf u$ through $\mathbf{w=x^T u}$ and the solutions should be identical, using the kernel or not.

But what if the kernel is not linear, what if the mapping in infinite dimensional so the column coefficients are impossible to represent with $\mathbf{w=\phi(\mathbf x)^T u}$, does it still makes sense to include an intercept term?

$\endgroup$
  • 2
    $\begingroup$ If the kernel is stationary, the intercept makes no difference by definition. $\endgroup$ – Sycorax says Reinstate Monica Aug 31 '16 at 1:15
3
$\begingroup$

Partial answer:

Focusing on SVMs for a while, I got to this reference (pointed by @DikranMarsupial in Bias term in support vector machine):

Poggio, T., Mukherjee, S., Rifkin, R., & Rakhlin, A. (2001). Verri, A. b. In Proceedings of the Conference on Uncertainty in Geometric Computations.

Excerpt:

This paper is devoted to answering the following questions: When should b be used? Is there a choice of using or not using b? What does the choice mean? Are the answers different for RNs (Regularization Networks) and SVMs? [...]

In their conclusion, they mention the use of a bias term is related to not privileging certain values for classification thresholds in SVMs. Also:

  • For infinite conditionally positive definite kernels the b term is de facto required allowing a natural interpretation of the optimizer.

  • For positive definite kernels the natural choice is without the b term, however it's possible to use one, actually leading to another kernel interpretation different of the one without it.

See that the minimizer is written including an explicit parameter b to be optimized.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.