4
$\begingroup$

Suppose that we have an array of 10x2 elements (features). Each of these features are two-dimensional. Something like this:

Array A:

0.001  0.56
0.045  0.12
0.546  0.54
0.123  0.12
1.435  0.01
1.234  0.01
.
.
.
1.654  0.12

Now, if I want to project this feature space in a higher dimension I need to implement a Kernel. Consider the linear kernel: $K(x,y) = (x' \times y +1)$.

I am confused about this $x$ and $y$. What are my $x$ and $y$ here? Do I need more data? These features are also supposed to belong in two different categories. Do I have to split array A into two different arrays each one contains samples from class A and B, and then do the dot products?

$\endgroup$
5
  • 1
    $\begingroup$ @MichaelDorner kernel functions define distance between data instances (rows), not features (columns). $x$ and $y$ must be instances, e.g. $x=[0.001, 0.56]$ and $y=[0.045, 0.12]$. $\endgroup$ Oct 4, 2015 at 10:56
  • 1
    $\begingroup$ I think the problem arises because you are confusing terminology here. Features don't belong to classes, instances do. Instances are individual objects (e.g. patients) and features are their respective characteristics (e.g. height, age, ...). A kernel function $K(\mathbf{x},\mathbf{y}) = \langle \varphi(\mathbf{x}), \varphi(\mathbf{y}) \rangle$ defines a dot product between $\mathbf{x}$ and $\mathbf{y}$ in some feature space constructed via the embedding function $\varphi(\cdot)$. $\endgroup$ Oct 4, 2015 at 10:58
  • $\begingroup$ @MarcClaesen Hmmm, thanks for your reply. I actually started implementing it with an input matrix 20x2. The result is a kernel matrix 20x20. If I get it well kernel is just a pairwise distance . $\endgroup$
    – Modium
    Oct 4, 2015 at 12:55
  • $\begingroup$ That's right, the kernel matrix consists of pairwise distances between all data instances (not features!). $\endgroup$ Oct 4, 2015 at 13:05
  • $\begingroup$ @MarcClaesen Thanks a lot, that actually clarified many things. I still don't get this though: Why they use this notation: K(xi,xj) , and then, they also use this notation: K(x,y) to describe the same thing, a kernel. I mean whats the difference if it's just the pairwise distance? $\endgroup$
    – Modium
    Oct 4, 2015 at 13:21

1 Answer 1

1
$\begingroup$

Let me explain by a simple example with 1D data $x$.

We want to classify + and -, which is quite simple for linear separable data:

enter image description here

Perfect, we are done, we have the purple decision boundary. But what to do with the following data?

enter image description here

You can not find one straight line separating these data set. So we have to find a mapping $\Phi$ from one dimensional $x$ to a two dimensional feature, let's call it $z$:

$\Phi(x) \rightarrow z, \quad z = \begin{pmatrix}x & x^2\end{pmatrix}^\top.$

Et voilà, we have no a feature space which is again linear separable, but no longer in 1D, but in 2D:

enter image description here

So you can see, there are no further data (features) involved. You do not need find further features (maybe you have to do, but not for the kernel trick), but find a mapping in higher space (in your case, if I got you right, from 2D two maybe 3D or more). And notice it is also possible to have a mapping from 2D to 2D, but transformed. This is one reason why finding and applying the right kernel is not trivial!

$\endgroup$
2
  • $\begingroup$ Thing is that my problem is not what kernel to apply, but HOW to apply it, given my dataset! This is what confuses me. I have seen this implementation of an RBF kernel-methods.net/matlab/kernels/rbf.m and they just use the dot product of the single input. Other implementations use a X and a Y input. This is what i don't get!! $\endgroup$
    – Modium
    Oct 4, 2015 at 9:25
  • 3
    $\begingroup$ Although this answer is correct, it is incomplete. Going from 1d to 2d, the data becomes linearly separable, but it also increases the dimensionality, which is not suitable in higher dimensions. The essence of the kernel trick is that, we can work in the lower dimension, without explicitly transforming the data to a higher dimension and get good separation between classes. $\endgroup$
    – kedarps
    Jan 11, 2018 at 2:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.