What exactly is the difference between a parametric and non-parametric model?

Question

I am confused with the definition of non-parametric model after reading this link Parametric vs Nonparametric Models and Answer comments of my another question.

Originally I thought "parametric vs non-parametric" means if we have distribution assumptions on the model (similar to parametric or non-parametric hypothesis testing). But both of the resources claim "parametric vs non-parametric" can be determined by if number of parameters in the model is depending on number of rows in the data matrix.

For kernel density estimation (non-parametric) such a definition can be applied. But under this definition how can a neural network be a non-parametric model, as the number of parameters in the model is depending on the neural network structure and not on the number of rows in the data matrix?

What exactly is the difference between parametric and a non-parametric model?

Answer 1

In a parametric model, the number of parameters is fixed with respect to the sample size. In a nonparametric model, the (effective) number of parameters can grow with the sample size.

In an OLS regression, the number of parameters will always be the length of $\beta$, plus one for the variance.

A neural net with fixed architecture and no weight decay would be a parametric model.

But if you have weight decay, then the value of the decay parameter selected by cross-validation will generally get smaller with more data. This can be interpreted as an increase in the effective number of parameters with increasing sample size.

Answer 2

I think that the word "effective" in the accepted answer should be deleted. Because due to the different number of effective parameters, as Aksakal pointed out, the accepted answer implies that Ridge and Lasso are non-parametric, but it doesn't seem to be true. Effective parameters (effective degrees of freedom) are characteristics of a learning algorithm, but not a model itself.

In a machine learning problem we have three things:

1. Data generation model. It describes our assumptions about the probabilistic distribution that generated our data. From mathematical statistics we know that data generation model can be parametric or non-parametric. As Glen_b pointed out, in parametric data generation model this distribution is defined by a fixed number of parameters. In nonparametric data generation model we don't have a distribution with a fixed number of parameters, we have milder assumptions about it, like continuity or symmetry.

2. Algorithm (hypothesis). It is a function $h: \mathcal{X} \to \mathcal{Y}$ from some hypothesis space $\mathcal{H}$. This function tries to predict the true target value on any sample $x$. Hypothesis space (model) can be parametric or non-parametric.
   In parametric hypothesis space (parametric model) every algorithm is uniquely defined by a fixed number of parameters (this number is the same for all algorithms from this space). Simple examples of parametric models are linear regression model: $\mathcal{H} = \{h(x;w,b) = \langle w, x \rangle + b \mid w \in \mathbb{R}^d, b \in \mathbb{R} \}$
   and linear (binary) classification model: $\mathcal{H} = \{h(x; w,b) = \mathrm{sign}(\langle w, x \rangle + b) \mid w \in \mathbb{R}^d, b \in \mathbb{R}\}$.
   In non-parametric models we can't describe every algorithm with the vector of parameters of the same constant size for all algorithms. Usually the number of parameters grows with the size of a training set. For example in the case of decision trees $\mathcal{H} = \{T(x; \Theta) \mid \Theta\}$, where $\Theta = \{J, \, \{R_j, \gamma_j\}_{j=1}^J\}$ is a vector of tree's parameters: $J$ is a number of terminal nodes in the tree, $R_j$ are subregions of the input space $\mathcal{X}$ corresponding to these nodes, and $\gamma_j$ are the predictions in these nodes. Trees can have different number of leaves and different number of internal nodes, so the space of decision trees is non-parametric (dimension of $\Theta$ will be different for different trees, if we train them on the datasets generated from the same distribution, that is, with the same number of features $d$, but with different number of observations in each dataset).

3. Method (learning algorithm). We can formalize it as a function $\mu: D \to \mathcal{H}$. It uses training set $D$ to fit some hypothesis $h \in \mathcal{H}$. If $\mathcal{H}$ is parametric we call $\mu$ parametric method. If $\mathcal{H}$ is non-parametric we call $\mu$ non-parametric method. For example, OLS, Ridge and Lasso are all parametric methods because they all use exactly the same parametric model $\mathcal{H} = \{h(x;w,b) = \langle w, x \rangle + b \mid w \in \mathbb{R}^d, b \in \mathbb{R} \}$ (as I said above, we call it "linear regression model"). Despite the fact that these methods have different number of effective parameters.

Keep in mind, that we can use parametric data generation model and non-parametric learning algorithm (or vice-versa). For example, we can have gaussian data generation model $Y = f(X) + \varepsilon$, where $\varepsilon \in \mathcal{N}(0, \sigma^2)$. Obviously, this is a parametric data generation model. But we can always fit non-parametric method (for example, kNN regression) on the training set $D$, generated by this model.
Similarly, we can fit parametric OLS method without any parametric assumptions about data generation process. In this case this method simply will not be equivalent to Maximum Likelihood Estimation.

---

There is a useful list of parametric and non-parametric methods from Murphy's MLaPP book:

Note that non-linear SVM (it is listed in the table) is a non-parametric method, whereas linear SVM (it is not listed in the table) is a parametric method because it fits linear classification model (linear classifier).

Answer 3

I think if the model is defined as a set of equations (can be a system of concurrent equations or a single one), and we learn its parameters, then is parametric. That includes differential equations, and even Navier-Stokes' equation. Models defined descriptively, regardless of how they are solved, fall into the category of nonparametric. Thus, OLS would be parametric, and even quantile regression, though belongs in the domain of nonparametric statistics, is a parametric model.

On the other hand, when we use SEM (structural equation modeling) to identify the model, it would be a nonparametric model - until we have solved the SEM. PCA would be parametric, because the equations are well defined, but CCA can be nonparametric, because we are looking for correlations across all variables, and if these are Spearman's correlations, we have a nonparametric model. With Pearson's correlations, we imply a parametric (linear) model. I think clustering algorithms would be nonparametric, unless we are looking for clusters of certain shape.

And then we have the nonparametric regression, which is nonparametric, and LOESS regression, which is parametric, but serves the same purpose: we define the equation and the window.

Answer 4

• Parametric model: assumes that the population can be adequately modeled by a probability distribution that has a fixed set of parameters.
• Non-parametric model: makes no assumptions about some probability distribution when modeling the data.