I have not understood this Wikipedia statement:

The difference between parametric model and non-parametric model is that the former has a fixed number of parameters, while the latter grows the number of parameters with the amount of training data.

How does the number of parameters in a non-parametric model increase with the amount of training data?

  • $\begingroup$ Hi. Did you get to see a practical implementation of this, where number of parameters are growing with new incoming data? It would be a huge help if you could point me to such a source. I have just started nonparametrics(HDP) and am struggling with relating the theory to practical. $\endgroup$ – rj dj Mar 25 at 16:22

Nonparametric methods don't specify something (maybe the distribution, maybe a relationship between two variables) with a fixed finite number of parameters; they're (potentially) infinite parametric.

Consider a loess curve (a form of nonparametric regression), for example; the parameters not only aren't explicit, if you attempt to count them, the number isn't even an integer.

On the other hand, you never need more than $n$ parameters to define $n$ observations; presumably in at least that sense* the number of parameters grows as you increase $n$.

*(though in other senses as well)

Consider, for example, a kernel density estimate using the usual AIMSE-optimal bandwidth selection; that has an effective number of parameters (e.g. as measured by Ye's generalized degrees of freedom) that grows as $n$ increases (but not in proportion to $n$).

However, since the statement is referenced (Murphy, Kevin (2012). Machine Learning: A Probabilistic Perspective. MIT), you should probably consult that work for the complete context.

  • $\begingroup$ I have been just introduced with non-parametric. In non-parametric method , is there any way to understand when only $1$ parameter is enough and when i need more than one parameter(2,3,etc) ? $\endgroup$ – time Feb 14 '15 at 6:42
  • $\begingroup$ I wouldn't read too much into the statement, nor worry over much about precisely how many parameters are required in any particular situation. The model is (at least potentially) infinite-parametric -- but these parameters - outside of the ones you're immediately testing, if any - need not be (and usually aren't) explicit. $\endgroup$ – Glen_b Feb 14 '15 at 6:47
  • $\begingroup$ Consider, for example, a Kolmogorov-Smirnov two sample test. No explicit parameters are involved. It doesn't matter whether the sample sizes are (10,15) or (85,120) -- the test works the same both times; the test statistic is the largest difference in ECDFs and while the model could be regarded as having $n_1+n_2$ parameters in that particular instance, that's of no particular interest. $\endgroup$ – Glen_b Feb 14 '15 at 6:54
  • $\begingroup$ It is written that in a parametric model all the parameters are in finite-dimensional parameter spaces. What do they mean by finite-dimensional parameter spaces? In normal distribution, $-\infty<\mu<\infty$. It doesn't range finitely. $\endgroup$ – time Feb 20 '15 at 13:11
  • 1
    $\begingroup$ It's not the domains of the parameters, but the dimension of the parameter vector itself. In the case of a univariate normal, the parameter vector is $(\mu,\sigma)^\top$, which is a vector of dimension 2. The boundedness or unboundedness of the values $\mu$ or $\sigma$ can take aren't under consideration. $\endgroup$ – Glen_b Feb 20 '15 at 14:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.