3
$\begingroup$

I was reading up the difference between parametric and non-parametric models on this site: https://sebastianraschka.com/faq/docs/parametric_vs_nonparametric.html

It says that linear SVM is parametric but RBF is non-parametric because "In the RBF kernel SVM, we construct the kernel matrix by computing the pair-wise distances between the training points, which makes it non-parametric."

The linear SVM uses the primal form which is :

enter image description here

whereas kernalized SVMs use only the dual form which is:

enter image description here

From the above equations, I don't see how linear SVM are parametric but Kernalized ones are non- parametric.

$\endgroup$

1 Answer 1

2
$\begingroup$

A model is a family of distributions or functions indexed by a parameter vector $\theta$. In parametric models, $\theta$ has a fixed, finite dimensionality. In nonparametric models, $\theta$ may be infinite dimensional. In practice, we deal with this infinite dimensional space by working with finite parameters whose number/dimensionality grows with the amount of training data.

This distinction can be seen in the SVM equations you wrote. Parameters of the linear SVM consist of the weight vector $w$ and bias $b$. So, there are $d+1$ parameters for a $d$-dimensional input space. This number is fixed, and does not depend on $n$, the number of training points. So, the linear SVM is parametric.

Parameters of the kernelized SVM include the full training set $\{(x_i, y_i)\}_{i=1}^n$, a weight $\alpha_i$ for each training point, and a bias $w_0$. The kernel function may include additional hyperparameters. Clearly, the number of parameters grows with the number of training points. So, the kernelized SVM is nonparametric.

$\endgroup$
1
  • $\begingroup$ good answer. on the other hand, the relevance of this distinction is very much questionable. $\endgroup$
    – carlo
    Aug 4, 2020 at 18:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.