11
$\begingroup$

I am reading the Wikipedia article on statistical models here, and I am somewhat perplexed as to the meaning of "non-parametric statistical models", specifically:

A statistical model is nonparametric if the parameter set $\Theta$ is infinite dimensional. A statistical model is semiparametric if it has both finite-dimensional and infinite-dimensional parameters. Formally, if $d$ is the dimension of $\Theta$ and $n$ is the number of samples, both semiparametric and nonparametric models have $d \rightarrow \infty$ as $n \rightarrow \infty$. If $d/n \rightarrow 0$ as $n \rightarrow \infty$, then the model is semiparametric; otherwise, the model is nonparametric.

I get that if the dimension, (I take that to literally mean, the number of parameters) of a model is finite, then this is a parametric model.

What does not make sense to me, is how we can have a statistical model that has an infinite number of parameters, such that we get to call it "non-parametric". Furthermore, even if that was the case, why the "non-", if in fact there are an infinite number of dimensions? Lastly, since I am coming at this from a machine-learning background, is there any difference between this "non-parametric statistical model" and say, "non-parametric machine learning models"? Finally, what might some concrete examples be of such "non-parametric infinite dimensional models" be?

$\endgroup$
  • 3
    $\begingroup$ Using another Wiki page (en.wikipedia.org/wiki/…) : 'Non-parametric models differ from parametric models in that the model structure is not specified a priori but is instead determined from data. The term non-parametric is not meant to imply that such models completely lack parameters but that the number and nature of the parameters are flexible and not fixed in advance.' so non-parametric is not having an infinite number of parameters but an unknown number of parameters. $\endgroup$ – Riff Aug 16 '16 at 7:21
  • $\begingroup$ I have a doubt. In Non-parametric models, we do define the model's structure a priori. For example, in Decision Trees (which is a Non-parametric model) we define max_depth. Then how can you say that this parameter is indeed learnt/determined from the data itself and not pre-determined by us? $\endgroup$ – Amarpreet Singh Oct 21 '18 at 17:31
4
$\begingroup$

As Johnnyboycurtis has answerd, non-parametric methods are those if it makes no assumption on the population distribution or sample size to generate a model.

A k-NN model is an example of a non-parametric model as it does not consider any assumptions to develop a model. A Naive Bayes or K-means is an example of parametric as it assumes a distribution for creating a model.

For instance, K-means assumes the following to develop a model All clusters are spherical (i.i.d. Gaussian). All axes have the same distribution and thus variance. All clusters are evenly sized.

As for k-NN, it uses the complete training set for prediction. It calculates the nearest neighbors from the test point for prediction. It assumes no distribution for creating a model.

For more info:

  1. http://pages.cs.wisc.edu/~jerryzhu/cs731/stat.pdf
  2. https://stats.stackexchange.com/a/133841/86202
  3. https://stats.stackexchange.com/a/133694/86202
$\endgroup$
  • $\begingroup$ Can you expand on this please? Why KNN is an example of a non-parametric, and why K-means might be? It's those details I am after, esp examples of non-parametric methods, and why / how they dont have an assumption on the population distribution. Thanks! $\endgroup$ – Creatron Aug 19 '16 at 6:43
  • $\begingroup$ @Creatron I have modified the answer for more explanation. $\endgroup$ – prashanth Aug 19 '16 at 9:42
3
$\begingroup$

So, I think you're missing a few points. First, and most importantly,

A statistical method is called non-parametric if it makes no assumption on the population distribution or sample size.

Here is a simple (applied) tutorial on some nonparmetric models: http://www.r-tutor.com/elementary-statistics/non-parametric-methods

A researcher may decide to use a nonparemtric model vs a parametric model, say, nonparamtric regression vs linear regression, is because the data violates assumptions held by the parametric model. Since you're coming from a ML background, I'll just assume you never learned the typical linear regression model assumptions. Here is a reference: https://statistics.laerd.com/spss-tutorials/linear-regression-using-spss-statistics.php

Violating assumptions can skew your parameter estimates, and ultimately increase the risk of invalid conclusions. A nonparametric model is more robust to outliers, nonlinear relationships, and does not depend on many population distribution assumptions, hence, can provide more trust worthy results when trying to make inferences or predictions.

For a quick tutorial on nonparametric regression, I recommend these slides: http://socserv.socsci.mcmaster.ca/jfox/Courses/Oxford-2005/slides-handout.pdf

$\endgroup$
  • $\begingroup$ Thanks for the links, I will go through them. One thing though, is how are we supposed to marry this with the "infinite number of parameters" that make up a "non-parametric" model? Thanks $\endgroup$ – Creatron Aug 16 '16 at 18:35
  • $\begingroup$ There's no citation for that "infinite number of parameters" so I cannot comment. I've never seen such a reference to the topic of nonparametric statistical model, so I would need to see a reference before I can provide an answer/interpretation. For now, I would worry about the assumptions to specific models vs a whole field. $\endgroup$ – Jon Aug 16 '16 at 18:45
  • $\begingroup$ The wikipedia article cited in my question refers to the infinite dimensionality. Literally: "A statistical model is non-parametric if the parameter set is infinite dimensional." What does this mean? This is what I am referring to. $\endgroup$ – Creatron Aug 16 '16 at 19:55
  • $\begingroup$ I know. But Wikipedia does not provide a citation for that statement. Can't trust something without a reference. $\endgroup$ – Jon Aug 16 '16 at 20:04
3
$\begingroup$

I'm currently taking a course on Machine learning, where we use the following definition of nonparametric models: "Nonparametric models grow in complexity with the size of the data".

Parametric model

To see what it mean let's have a look at linear regression, a parametric model: There we try to predict a function parametrized in $ w \in ℝ ^d $: $$ f(x) = w^Tx $$ The dimensionality of w is independent of the number of observations, or the size of your data.

Nonparametric models

Instead kernel regression tries to predict the following function: $$ f(x) = \sum_{i=1}^n \alpha_i k(x_i, x) $$ where we have $n$ data points, $\alpha_i$ are the weights and $k(x_i, x)$ is the kernel function. Here the number of parameters $\alpha_i$ is dependent on the number of data points $n$.

The same is true for the kernelized perceptron: $$ f(x) = sign( \sum_{i=1}^n \alpha_i y_i k(x_i,x))) $$

Let's come back to your definition and say d was the number of $\alpha_i$. If we let $ n \to \infty $ then $d \to \infty$. That's exactly what the wikipedia definition asks for.

I took the kernel regression function from my lecture slides and the kernelized perceptron function from wikipedia: https://en.wikipedia.org/wiki/Kernel_method

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.