Formal definitions for nonparametric and parametric models

Question

I am looking for formal definitions for what are parametric and nonparametric models? The ones that we can find there and there are often hand-waving...

For example, would you say that - a SVM (support vector machine) is parametric or non-parametric? - a neural network?

Answer 1

Mark J. Schervish's Theory of Statistics (1995) puts it like this (p. 1):

Most paradigms for statistical inference make at least some use of the following structure. We suppose that some random variables $X_1, ..., X_n$ all have the same distribution [i.e., their induced probability measures are all equal], but we may be unwilling to say what that distribution is. Instead, we create a collection of distributions called a parametric family and denoted $P_0$. For example, $P_0$ might consist of all normal distributions, or just those normal distributions with variance 1, or all binomial distributions, or all Poisson distributions, and so forth. Each of these cases has the property that the collection of distributions can be indexed by a finite-dimensional real quantity, which is commonly called a parameter. For example, if the parametric family is all normal distributions, then the parameter can be denoted $Θ = (M, E)$, where $M$ stands for the mean and $E$ stands for the standard deviation. The set of all possible values of the parameter is called the parameter space and is often denoted by $Ω$.

Schervish's terminology is a little idiosyncratic in that he calls the vector of real numbers determining the distribution "the parameter", singular, as opposed to the more common practice of referring to the individual real numbers as "parameters".

Notice Schervish's qualification "finite-dimensional". Parametric families with infinite-dimensional parameter spaces do see use, but the resulting model is (perhaps confusingly) called nonparametric. In my opinion, "nonparametric" is also a fair description of any method that doesn't fit into the parametric-modeling framework at all, such as nearest-neighbors classification, but one could argue instead that the terms "parametric" and "nonparametric" are simply inapplicable.

Answer 2

• Parametric (model): data which can be described by a finite number of parameters (for example gaussian with both unknown mean and variance).
• Nonparametric (model): data which need an infinite number of parameters to be described (for example bounded probability distributions, continuous regression functions...)
• Parametric/nonparametric algorithms: algorithms devised to work for parametric/nonparametric models.

Basic SVM only works for linear data, so it's a parametric algorithm. More advanced SVM work for more complicated data. If your SVM works for nonparametric data (for example regression functions only assumed to be continuous), then it is nonparametric.