43
$\begingroup$

I am confused with the definition of non-parametric model after reading this link Parametric vs Nonparametric Models and Answer comments of my another question.

Originally I thought "parametric vs non-parametric" means if we have distribution assumptions on the model (similar to parametric or non-parametric hypothesis testing). But both of the resources claim "parametric vs non-parametric" can be determined by if number of parameters in the model is depending on number of rows in the data matrix.

For kernel density estimation (non-parametric) such a definition can be applied. But under this definition how can a neural network be a non-parametric model, as the number of parameters in the model is depending on the neural network structure and not on the number of rows in the data matrix?

What exactly is the difference between parametric and a non-parametric model?

$\endgroup$
4
  • 4
    $\begingroup$ Note that "nonparametric" in relation to distributional models (as in your reference to hypothesis tests) relates to the number of parameters used to define the distribution ("parametric' = defined by a fixed number of parameters; nonparametric methods don't have a distribution with a fixed number of parameters -- they tend to have milder assumptions, like continuity or symmetry) $\endgroup$
    – Glen_b
    Mar 20, 2017 at 22:43
  • $\begingroup$ My opinion: stick to your definition. It's a systematic definition, as definitions should be. The other one is shaky: you first need to define the "number of effective parameters" of an algorithm. But I have always seen this quantity defined on a case by case basis (i.e. you have one definition for a linear regression, one for nearest neighbour, one for neural networks..). So unless someone can offer a general, systematic definition of the effective number of parameters, I can't really take this definition seriously. $\endgroup$
    – Adrien
    Mar 21, 2017 at 10:51
  • 3
    $\begingroup$ Found below link which has good explanation of parametric machine learning algorithms and non-parametric machine learning algorithms. machinelearningmastery.com/… $\endgroup$
    – Satya
    Jan 13, 2018 at 10:40
  • 1
    $\begingroup$ The naming is really bad: parametric models should better be called "models with fixed number of parameters", non-parametric models should better be called "models with unbound number of parameters". $\endgroup$
    – asmaier
    Nov 16, 2021 at 10:35

4 Answers 4

37
$\begingroup$

In a parametric model, the number of parameters is fixed with respect to the sample size. In a nonparametric model, the (effective) number of parameters can grow with the sample size.

In an OLS regression, the number of parameters will always be the length of $\beta$, plus one for the variance.

A neural net with fixed architecture and no weight decay would be a parametric model.

But if you have weight decay, then the value of the decay parameter selected by cross-validation will generally get smaller with more data. This can be interpreted as an increase in the effective number of parameters with increasing sample size.

$\endgroup$
6
  • 2
    $\begingroup$ Surely though the weight decay parameter is still a single additional parameter and doesn't (unless I'm mistaken) change the structure of the network. How can it be interpreted as an increase in the number of parameters as the sample size increases? $\endgroup$ Mar 20, 2017 at 15:13
  • 4
    $\begingroup$ The weight decay is a hyperparameter. Read here about effective degrees of freedom in regularization: statweb.stanford.edu/~tibs/sta305files/Rudyregularization.pdf. While neural nets aren't linear, the weight decay performs the same function as a quadratic penalty in these models. $\endgroup$ Mar 20, 2017 at 15:17
  • 2
    $\begingroup$ I (of course) agree with the intuition of effective parameters, but I don't agree with using this notion to define parametric/nonparametric, see my comment to the question. $\endgroup$
    – Adrien
    Mar 21, 2017 at 10:53
  • 3
    $\begingroup$ I wonder if this explanation might be mixing up models and algorithms. The learning algorithm is a way to obtain a model given a dataset, and is affected by regularization. A particular model output by the learning algorithm is a function. In the case of neural nets with a fixed architecture, (even where weight decay is used during learning), all functions can be indexed by a constant, finite number of parameters, independent of the dataset size. Compare this to something like kernel methods, where this is not the case. $\endgroup$
    – user20160
    Jan 8, 2018 at 7:53
  • 6
    $\begingroup$ I’ve seen this explanation before and didn’t like it. This way I can call an ordinary least squares with shrinkage a Nonparametric method because the “effective “ parameters can be fewer than coefficients. I think it is not a helpful categorization as it blurs the Line between truly Nonparametric methods $\endgroup$
    – Aksakal
    Nov 17, 2018 at 1:25
14
$\begingroup$

I think that the word "effective" in the accepted answer should be deleted. Because due to the different number of effective parameters, as Aksakal pointed out, the accepted answer implies that Ridge and Lasso are non-parametric, but it doesn't seem to be true. Effective parameters (effective degrees of freedom) are characteristics of a learning algorithm, but not a model itself.

In a machine learning problem we have three things:

  1. Data generation model. It describes our assumptions about the probabilistic distribution that generated our data. From mathematical statistics we know that data generation model can be parametric or non-parametric. As Glen_b pointed out, in parametric data generation model this distribution is defined by a fixed number of parameters. In nonparametric data generation model we don't have a distribution with a fixed number of parameters, we have milder assumptions about it, like continuity or symmetry.

  2. Algorithm (hypothesis). It is a function $h: \mathcal{X} \to \mathcal{Y}$ from some hypothesis space $\mathcal{H}$. This function tries to predict the true target value on any sample $x$. Hypothesis space (model) can be parametric or non-parametric.
    In parametric hypothesis space (parametric model) every algorithm is uniquely defined by a fixed number of parameters (this number is the same for all algorithms from this space). Simple examples of parametric models are linear regression model: $\mathcal{H} = \{h(x;w,b) = \langle w, x \rangle + b \mid w \in \mathbb{R}^d, b \in \mathbb{R} \}$
    and linear (binary) classification model: $\mathcal{H} = \{h(x; w,b) = \mathrm{sign}(\langle w, x \rangle + b) \mid w \in \mathbb{R}^d, b \in \mathbb{R}\}$.
    In non-parametric models we can't describe every algorithm with the vector of parameters of the same constant size for all algorithms. Usually the number of parameters grows with the size of a training set. For example in the case of decision trees $\mathcal{H} = \{T(x; \Theta) \mid \Theta\}$, where $\Theta = \{J, \, \{R_j, \gamma_j\}_{j=1}^J\}$ is a vector of tree's parameters: $J$ is a number of terminal nodes in the tree, $R_j$ are subregions of the input space $\mathcal{X}$ corresponding to these nodes, and $\gamma_j$ are the predictions in these nodes. Trees can have different number of leaves and different number of internal nodes, so the space of decision trees is non-parametric (dimension of $\Theta$ will be different for different trees, if we train them on the datasets generated from the same distribution, that is, with the same number of features $d$, but with different number of observations in each dataset).

  3. Method (learning algorithm). We can formalize it as a function $\mu: D \to \mathcal{H}$. It uses training set $D$ to fit some hypothesis $h \in \mathcal{H}$. If $\mathcal{H}$ is parametric we call $\mu$ parametric method. If $\mathcal{H}$ is non-parametric we call $\mu$ non-parametric method. For example, OLS, Ridge and Lasso are all parametric methods because they all use exactly the same parametric model $\mathcal{H} = \{h(x;w,b) = \langle w, x \rangle + b \mid w \in \mathbb{R}^d, b \in \mathbb{R} \}$ (as I said above, we call it "linear regression model"). Despite the fact that these methods have different number of effective parameters.

Keep in mind, that we can use parametric data generation model and non-parametric learning algorithm (or vice-versa). For example, we can have gaussian data generation model $Y = f(X) + \varepsilon$, where $\varepsilon \in \mathcal{N}(0, \sigma^2)$. Obviously, this is a parametric data generation model. But we can always fit non-parametric method (for example, kNN regression) on the training set $D$, generated by this model.
Similarly, we can fit parametric OLS method without any parametric assumptions about data generation process. In this case this method simply will not be equivalent to Maximum Likelihood Estimation.


There is a useful list of parametric and non-parametric methods from Murphy's MLaPP book: enter image description here

Note that non-linear SVM (it is listed in the table) is a non-parametric method, whereas linear SVM (it is not listed in the table) is a parametric method because it fits linear classification model (linear classifier).

$\endgroup$
1
  • 4
    $\begingroup$ This blogpost by Sebastian Raschka confirms that linear SVM is a parametric method, whereas kernel SVM is non-parametric. It also contains some good thoughts on the subject of parametric vs non-parametric models, which I really like. Particularly, it has the following quote from the book written in 1962: "A precise and universally acceptable definition of the term ‘nonparametric’ is not presently available". I think this is still true nowadays. $\endgroup$
    – Rodvi
    Mar 5, 2021 at 14:44
0
$\begingroup$

I think if the model is defined as a set of equations (can be a system of concurrent equations or a single one), and we learn its parameters, then is parametric. That includes differential equations, and even Navier-Stokes' equation. Models defined descriptively, regardless of how they are solved, fall into the category of nonparametric. Thus, OLS would be parametric, and even quantile regression, though belongs in the domain of nonparametric statistics, is a parametric model.

On the other hand, when we use SEM (structural equation modeling) to identify the model, it would be a nonparametric model - until we have solved the SEM. PCA would be parametric, because the equations are well defined, but CCA can be nonparametric, because we are looking for correlations across all variables, and if these are Spearman's correlations, we have a nonparametric model. With Pearson's correlations, we imply a parametric (linear) model. I think clustering algorithms would be nonparametric, unless we are looking for clusters of certain shape.

And then we have the nonparametric regression, which is nonparametric, and LOESS regression, which is parametric, but serves the same purpose: we define the equation and the window.

$\endgroup$
3
  • 4
    $\begingroup$ Your descriptions are rather vague and seem to be at odds with the standard statistical meaning of "parametric" and "nonparametric." In particular, you have taken an unusual position concerning some particular techniques, such as LOESS, which is generally considered nonparametric: see en.wikipedia.org/wiki/Local_regression for instance. $\endgroup$
    – whuber
    Oct 31, 2018 at 1:34
  • $\begingroup$ @whuber thanks for the link! You are correct: LOESS is considered nonparametric. Which is rather counterintuitive to me. What about exponential smoothing? Is it nonparametric because the weight of every point is different? Or is it parametric because the alpha is the same for the entire time series? $\endgroup$
    – AlexG
    Nov 1, 2018 at 4:51
  • $\begingroup$ The parameters in parametric situations do not necessarily count a bunch of numbers. They refer to how one must describe a family of statistical models. For instance, when a procedure fits a single value to data (perhaps by cross-validation, perhaps by other means) but assumes only that the data are a random sample from any distribution, that procedure is non-parametric. $\endgroup$
    – whuber
    Nov 1, 2018 at 12:21
0
$\begingroup$
  • Parametric model: assumes that the population can be adequately modeled by a probability distribution that has a fixed set of parameters.
  • Non-parametric model: makes no assumptions about some probability distribution when modeling the data.
$\endgroup$
9
  • $\begingroup$ SVM is a parametric model, but it is not a probabilistic model at all (at least in its classic definition). So, SVM doesn't assume any probabilistic distribution. I think you talk about parametric / non-parametric data generation model, but the author asks about parametric / non-parametric algorithms (such as neural net, which he noted) and methods, and it is not the same thing. $\endgroup$
    – Rodvi
    Mar 5, 2021 at 12:04
  • $\begingroup$ * I mean linear SVM. In your classification, it should be a non-parametric model, but in the textbooks and other sources this model is usually considered to be parametric. $\endgroup$
    – Rodvi
    Mar 5, 2021 at 12:18
  • $\begingroup$ Could you provide me with reliable sources that state that LSVM is a parametric model which doesn't assume any probability distribution? $\endgroup$
    – crocefisso
    Mar 5, 2021 at 12:39
  • 2
    $\begingroup$ "Makes no assumptions" is sometimes stated by non-statisticians, but is so limited that it doesn't apply to almost every non-parametric procedure that has been published: almost all make some kind of restrictive assumption, such as that the underlying distribution is continuous. The Wikipedia article is poorly worded: the key phrase "makes no assumptions about a parametric distribution" should be taken literally to mean "doesn't assume a model with a finite set of parameters." That is not the same as "makes no assumptions." $\endgroup$
    – whuber
    Mar 5, 2021 at 15:46
  • 1
    $\begingroup$ There's no point to that, because good and correct answers have already been posted in this thread. $\endgroup$
    – whuber
    Mar 5, 2021 at 19:42

Not the answer you're looking for? Browse other questions tagged or ask your own question.