I am relatively new to non-parametric models, such as the Hierarchical Dirichlet Process, Pitman-Yor processes, etc. but I have read that non-parametric methods are data-driven, and that they can be more robust than their parametric counter-parts. I am looking for evidence and an exposition of why this is the case.

Perhaps more concisely, what are the benefits and drawbacks of non-parametric models (and what evidence there is of this). More informally, can it be argued that there is a reasonable trend to move towards non-parametric models in the machine learning / statistics literature?

  • $\begingroup$ With the assumption that "non parametric" implies a model NOT built using a statistical distribution ( rather than a model which has no parameters at all ) then many popular threads of machine learning research have never included parametric models, e.g. Multi Layer Perceptrons, Decision Trees and Forests, K-Nearest Neighbours, Boosting etc so it seems hard to characterise these methods as trending toward non-parametricity. Maybe this boils down to debate over whether Statistical Pattern Recognition, Machine Learning, Statistics, Econometrics, Operations Research, Computer Science is in vogue. $\endgroup$ – image_doctor Dec 10 '12 at 17:50
  • 1
    $\begingroup$ Bayesian nonparametric models refer to a class of Bayesian models whose priors are defined by stochastic processes. As the OP mentions, well known examples are Dirichlet Process, Hierarchical Dirichlet Process, Guassian Process, and Indian Buffet Process. See here for a good introduction. $\endgroup$ – jerad Dec 10 '12 at 19:20
  • 1
    $\begingroup$ @image_doctor, I must respectfully disagree. In the past 5 years, there's been a a great deal of interest in these models. There's at least a dozen such papers accepted to NIPS this year, and similar numbers for the past several years. Similarly, over the last few years there's been regular workshops and conferences devoted to Bayesian Nonparametrics. See here for one held at NIPS last year that discusses their recent trend. $\endgroup$ – jerad Dec 10 '12 at 23:27
  • 1
    $\begingroup$ @jerad I don't disagree that they are in vogue in the research world, but I would take a different view on whether they predominate in the tool box of practically applied solutions to real world problems. Plucking a figure from the air, probably more than 90% of machine learning techniques applied are still based on established methods more than 10 years old. $\endgroup$ – image_doctor Dec 11 '12 at 9:20
  • 1
    $\begingroup$ @image_doctor, Okay, well on that point we also agree. $\endgroup$ – jerad Dec 12 '12 at 15:53

In parametric statistics, it is assumed that the data follow a distribution that is known up to the value of a fixed (and small) number of values -- the parameters. Non-parametric methods make fewer assumptions. One might posit, say, that the data are IID from a continuous (or discrete) distribution. Estimation proceeds from there.

Strictly speaking, both methods are data driven, in the sense that something must be estimated from the data. Non-parametric methods make fewer assumptions about the underlying function. Those assumptions that are made -- such as independence or identity of distribution -- need to hold for the methods to be valid. Non-parametrics are not some sort of safe, "anything goes" type of thing that you can do when the data are a bunch of garbage.

Non-parametric methods are not necessarily robust. Robustness implies that sensible results will obtain in the presence of contaminated data or highly unusual values. A bootstrap method, for example, can be highly influenced by a few outliers. A trimmed mean, on the other hand, under the assumption of normal variates, would be robust but parametric.

If the distribution is known (or knowable) up to its parameters, then parametric estimation is more efficient. That probably doesn't matter if you have a large amount of data -- which you would if you are delving into Gaussian processes and the other processes mentioned by jerad.

Lehmann's 1975 book is the classic reference for the original work done on non-parametric methods in the context of classic statistical theory, which might answer some of your foundational questions. Since then, of course, a lot of work has been done on the bootstrap (one way to go) and numerous machine learning ventures (another direction).

The entire concept of robustness belongs to classical, statistical theory, I believe. Machine learning people are more concerned with stability. It's a very different framework.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.