When using RBF kernel I think feature space is infinite dimensional space. With infinite dimensional features, I believe any training set can be classified. So I'm wondering why training error > 0 even then? Does smoothing factor occurs that?


I forgot to mention about the identical feature vectors and the opposite label. Please ignore that case.

  • 2
    $\begingroup$ What Happens if two examples have identical feature vectors but opposite labels? $\endgroup$ – Sycorax May 14 '16 at 4:50
  • $\begingroup$ What if you have 1000 irrelevant features and one moderately relevant feature? What if all features are irrelevant? $\endgroup$ – Sycorax May 14 '16 at 8:46
  • $\begingroup$ @C11H17N2O2SNa if all features are irrelevant, we can still get 0 error by sort of "memorizing" the mapping. i'm not sure is that doable with RBF kernel? to overfit everything? $\endgroup$ – dontloo May 14 '16 at 10:13

"infinite dimensional features" is not the same as "all possible features".

the infinite feature space of rbf consists of a $exp(-x^2)$ term and infinite $\sqrt{\frac{2^k}{k!}}x^k$ terms with $k=0$ to infinity, according to the Taylor series for the exponential function.

if the data is only separable with features other than those, the training error will be greater than 0. and yes the smoothing factor also matters.

moreover if there happen to be two identical samples with different labels, the data will be non-separable in any case.

  • $\begingroup$ Thank you. Is it possible to explain what kind of feature set can not be separable with RBF kernel? Please ignore the identical vectors and the opposite labels. $\endgroup$ – rkjt50r983 May 14 '16 at 6:44
  • $\begingroup$ @rkjt50r983 you're welcome, sorry I am not sure about this, maybe you could open up a new question. :) $\endgroup$ – dontloo May 14 '16 at 10:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.