I was told that these two are non-parametric models. But I can't figure out why, especially for KNN.
Could anyone answer my questions?
$\begingroup$
$\endgroup$
1
-
3$\begingroup$ Non-parametric does not mean that a model has no parameter but instead implies that the number and nature of those parameters are not fixed in advance. So KNN is non-parametric because it does not have a pre-defined way to describe your classes (whereas logistic regression will 'follow' your model equation whatever the training data is). $\endgroup$– RiffCommented Sep 27, 2016 at 8:03
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
kNN and SVM are not models, they are algorithms.
Your model is what you are going to assume for you data. For example, :
- Gaussian with unknown mean but fixed variance.
- Gaussian with both unknown mean and variance.
- bounded.
- continuous.
In the first two cases, the model is parametric because it is fully known except for a finite number of parameters (one -the mean- in the first case, and two -the mean and the variance- in the second case). In the third and fourth case, the model is nonparametric because you need an infinite number of parameters to specify the data (you can't describe the set of all bounded or continuous data with a finite number of parameters).
In a parametric model, you have a lot more information and you can use very specific algorithms (in our example algorithms which only work for gaussian data). In the nonparametric setting, you have to use more general algorithms, since you assume way less on the data. Nothing prevents you of using general nonparametric algorithms for parametric models, they would work, but you can generally do better by using specific algorithms.
kNN (even defined with gaussian weights) is a nonparametric algorithm devised to work for nonparametric models, i.e. very general models. SVM are more complicated to label. Basic SVM are linear classifiers, and as such parametric algorithms. Advanced SVM can work for nonlinear data, and if you have a SVM working for data not constrained to be in a family described by a finite number of parameters, then it is nonparametric.
PS: Like Nicolas stated, all theses algorithms, parametric or nonparametric (which refer to the model they work for) also have parameters you have to chose (for kNN, $k$, the number of nearest neighbors).
