8
$\begingroup$

I am confused with the definition of non-parametric model after reading this link Parametric vs Nonparametric Models and Answer comments of my another question.

Originally I thought "parametric vs non-parametric" means if we have distribution assumptions on the model (similar to parametric or non-parametric hypothesis testing). But both of the resources claim "parametric vs non-parametric" can be determined by if number of parameters in the model is depending on number of rows in the data matrix.

For kernel density estimation (non-parametric) such a definition can be applied. But under this definition how can a neural network be a non-parametric model, as the number of parameters in the model is depending on the neural network structure and not on the number of rows in the data matrix?

What exactly is the difference between parametric and a non-parametric model?

$\endgroup$
  • 3
    $\begingroup$ Note that "nonparametric" in relation to distributional models (as in your reference to hypothesis tests) relates to the number of parameters used to define the distribution ("parametric' = defined by a fixed number of parameters; nonparametric methods don't have a distribution with a fixed number of parameters -- they tend to have milder assumptions, like continuity or symmetry) $\endgroup$ – Glen_b Mar 20 '17 at 22:43
  • $\begingroup$ My opinion: stick to your definition. It's a systematic definition, as definitions should be. The other one is shaky: you first need to define the "number of effective parameters" of an algorithm. But I have always seen this quantity defined on a case by case basis (i.e. you have one definition for a linear regression, one for nearest neighbour, one for neural networks..). So unless someone can offer a general, systematic definition of the effective number of parameters, I can't really take this definition seriously. $\endgroup$ – Adrien Mar 21 '17 at 10:51
  • 1
    $\begingroup$ Found below link which has good explanation of parametric machine learning algorithms and non-parametric machine learning algorithms. machinelearningmastery.com/… $\endgroup$ – Satya Jan 13 '18 at 10:40
8
$\begingroup$

In a parametric model, the number of parameters is fixed with respect to the sample size. In a nonparametric model, the (effective) number of parameters can grow with the sample size.

In an OLS regression, the number of parameters will always be the length of $\beta$, plus one for the variance.

A neural net with fixed architecture and no weight decay would be a parametric model.

But if you have weight decay, then the value of the decay parameter selected by cross-validation will generally get smaller with more data. This can be interpreted as an increase in the effective number of parameters with increasing sample size.

$\endgroup$
  • 1
    $\begingroup$ Surely though the weight decay parameter is still a single additional parameter and doesn't (unless I'm mistaken) change the structure of the network. How can it be interpreted as an increase in the number of parameters as the sample size increases? $\endgroup$ – Morgan Ball Mar 20 '17 at 15:13
  • 1
    $\begingroup$ The weight decay is a hyperparameter. Read here about effective degrees of freedom in regularization: statweb.stanford.edu/~tibs/sta305files/Rudyregularization.pdf. While neural nets aren't linear, the weight decay performs the same function as a quadratic penalty in these models. $\endgroup$ – generic_user Mar 20 '17 at 15:17
  • $\begingroup$ I (of course) agree with the intuition of effective parameters, but I don't agree with using this notion to define parametric/nonparametric, see my comment to the question. $\endgroup$ – Adrien Mar 21 '17 at 10:53
  • $\begingroup$ Yeah I see your point. But I suppose that reasonable people can disagree about whether the shakyness of a definition renders it an unhelpful definition, ceteris paribus. $\endgroup$ – generic_user Mar 21 '17 at 13:20
  • $\begingroup$ I wonder if this explanation might be mixing up models and algorithms. The learning algorithm is a way to obtain a model given a dataset, and is affected by regularization. A particular model output by the learning algorithm is a function. In the case of neural nets with a fixed architecture, (even where weight decay is used during learning), all functions can be indexed by a constant, finite number of parameters, independent of the dataset size. Compare this to something like kernel methods, where this is not the case. $\endgroup$ – user20160 Jan 8 '18 at 7:53
0
$\begingroup$

Parametric model can be outputted using an equation, like logistic regression model, $logodds(G)=int + ax_1 + bx_2 + ...$. The non-parametric model are black box algorithms like random forest, decision tree. There is no equation that can describe the relation of attributes behind the model.

$\endgroup$
  • 1
    $\begingroup$ I can also write equations for kernel estimation methods, which is non-parametric. $\endgroup$ – SmallChess Mar 20 '17 at 15:20
  • 1
    $\begingroup$ wrong - you can write explicit and simple equations for the predictive mean and the predictive variance of Gaussian Processes, which are one of the most common nonparametric regression methods, and for many other nonparametric regression methods. $\endgroup$ – DeltaIV Mar 20 '17 at 17:20
0
$\begingroup$

I think if the model is defined as a set of equations (can be a system of concurrent equations or a single one), and we learn its parameters, then is parametric. That includes differential equations, and even Navier-Stokes' equation. Models defined descriptively, regardless of how they are solved, fall into the category of nonparametric. Thus, OLS would be parametric, and even quantile regression, though belongs in the domain of nonparametric statistics, is a parametric model.

On the other hand, when we use SEM (structural equation modeling) to identify the model, it would be a nonparametric model - until we have solved the SEM. PCA would be parametric, because the equations are well defined, but CCA can be nonparametric, because we are looking for correlations across all variables, and if these are Spearman's correlations, we have a nonparametric model. With Pearson's correlations, we imply a parametric (linear) model. I think clustering algorithms would be nonparametric, unless we are looking for clusters of certain shape.

And then we have the nonparametric regression, which is nonparametric, and LOESS regression, which is parametric, but serves the same purpose: we define the equation and the window.

$\endgroup$
  • 3
    $\begingroup$ Your descriptions are rather vague and seem to be at odds with the standard statistical meaning of "parametric" and "nonparametric." In particular, you have taken an unusual position concerning some particular techniques, such as LOESS, which is generally considered nonparametric: see en.wikipedia.org/wiki/Local_regression for instance. $\endgroup$ – whuber Oct 31 '18 at 1:34
  • $\begingroup$ @whuber thanks for the link! You are correct: LOESS is considered nonparametric. Which is rather counterintuitive to me. What about exponential smoothing? Is it nonparametric because the weight of every point is different? Or is it parametric because the alpha is the same for the entire time series? $\endgroup$ – AlexG Nov 1 '18 at 4:51
  • $\begingroup$ The parameters in parametric situations do not necessarily count a bunch of numbers. They refer to how one must describe a family of statistical models. For instance, when a procedure fits a single value to data (perhaps by cross-validation, perhaps by other means) but assumes only that the data are a random sample from any distribution, that procedure is non-parametric. $\endgroup$ – whuber Nov 1 '18 at 12:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.