3
$\begingroup$

I have read every explanation out there on this but nobody seems capable of explaining this in a way that I am able to understand. For an SVM RBF Kernel we often say that: enter image description here

But what does x and x' represent here? Let's say my input data x is of shape 5x100 (5 features aka dimensions aka columns) and 100 rows, and output y is 1x100 rows, then what is x and x' in relation to this?

|cite|improve this question
$\endgroup$
  • $\begingroup$ This looks very similar to this question. $x$ and $x'$ are individual measurements (so 5x1 in your example), and $K(x,x')$ gives the covariance between the observations for the corresponding measurements. $\endgroup$ – combo Aug 29 '17 at 22:50
  • $\begingroup$ The link you gave is to an Gaussian Process RBF not an SVM RBF Kernel. None of the explanations for SVM RBF ever mention the word co-variance hence my confusion (en.wikipedia.org/wiki/Radial_basis_function_kernel). If K(x,x') is the covariance , does that mean x' is the transpose of x? $\endgroup$ – user1761806 Aug 29 '17 at 23:15
  • $\begingroup$ As a further example of why I'm genuinally confused, see this. Here co-variance is not mentioned, instead it is 'test' data. Then look at this here nothing is mentioned either about covariance or about test data. $\endgroup$ – user1761806 Aug 29 '17 at 23:28
  • $\begingroup$ Prime notation in this case just means "different". The kernel is a measure of similarity (e.g. an inner product) - in a Gaussian process setting it is covariance between samples, in the SVM setting it is similarity between samples (the basic math/kernel trick is the same in either case). The first link you give uses the kernel to quantify the similarity between the train and test features; The second certainly mentions test data, and also discusses the kernel as a similarity metric. I think thoroughly reading this tutorial would help you. $\endgroup$ – combo Aug 30 '17 at 0:43
  • 1
    $\begingroup$ Awesome - thanks. Yes I finally got to the same conclusion after I read this, and this. I did read Andrew Ngs notes (pg. 14-16 are the key bits for anyone reading it) but did not find it clearly addressing the above point (couldn't tell if i,j were samples or dimensions). Anyway thanks for your help -- if you post it as an answer I can award the points. Much appreciated! $\endgroup$ – user1761806 Aug 30 '17 at 1:19
2
$\begingroup$

The kernel function is a measure of similarity between two sets of features. So in this case, $x'$ and $x$ will both be $5\times 1$ feature vectors (not necessarily the same). $K(x,x')$ is a scalar that represents the similarity between $x$ and $x'$, and the kernel matrix $[K(x,x')]_{x\in X, x'\in X}$ is a $100\times 100$ matrix which represents the pairwise similarities.

The kernel function can be thought of as a cheap way of computing an infinite dimensional inner product - this 'kernel trick' is described in more detail in these notes. This lets the algorithm learn arbitrarily complex functions (though it may take an infinite number of samples for it to learn).

|cite|improve this answer
$\endgroup$
1
$\begingroup$

Here's my attempt at giving a non-mathy explanation for the RBF kernel. The kernel function gives you the distance between $x$ and $x'$. It's not like the regular Euclidean distance though. I imagine living in an RBF kernel world like walking around in a little bubble where anything what is within arms reach of me is just like normal Euclidean distances but anything farther than that (outside the bubble) is warped to be extremely far The way this works with the math is $\sigma$ is like the size of the bubble and the $\exp$ function is what pushes everything to be really far away if it is not already close.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ I don't think it's correct to say that the kernel function gives a distance. It returns values that are higher for nearby points, with a maximum when $x=x'$. This is opposite to how distance behaves. $\endgroup$ – user20160 Aug 30 '17 at 14:22
  • $\begingroup$ You're right. But you could just invert it and then you would have something that behaves like a distance. $\endgroup$ – Aaron Aug 30 '17 at 19:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.