The case for non-parametric models I am relatively new to non-parametric models, such as the Hierarchical Dirichlet Process, Pitman-Yor processes, etc. but I have read that non-parametric methods are data-driven, and that they can be more robust than their parametric counter-parts. I am looking for evidence and an exposition of why this is the case.
Perhaps more concisely, what are the benefits and drawbacks of non-parametric models (and what evidence there is of this). More informally, can it be argued that there is a reasonable trend to move towards non-parametric models in the machine learning / statistics literature?
 A: In parametric statistics, it is assumed that the data follow a distribution that is known up to the value of a fixed (and small) number of values -- the parameters. Non-parametric methods make fewer assumptions. One might posit, say, that the data are IID from a continuous (or discrete) distribution. Estimation proceeds from there. 
Strictly speaking, both methods are data driven, in the sense that something must be estimated from the data. Non-parametric methods make fewer assumptions about the underlying function. Those assumptions that are made -- such as independence or identity of distribution -- need to hold for the methods to be valid. Non-parametrics are not some sort of safe, "anything goes" type of thing that you can do when the data are a bunch of garbage. 
Non-parametric methods are not necessarily robust. Robustness implies that sensible results will obtain in the presence of contaminated data or highly unusual values. A bootstrap method, for example, can be highly influenced by a few outliers. A trimmed mean, on the other hand, under the assumption of normal variates, would be robust but parametric.
If the distribution is known (or knowable) up to its parameters, then parametric estimation is more efficient. That probably doesn't matter if you have a large amount of data -- which you would if you are delving into Gaussian processes and the other processes mentioned by jerad.
Lehmann's 1975 book is the classic reference for the original work done on non-parametric methods in the context of classic statistical theory, which might answer some of your foundational questions. Since then, of course, a lot of work has been done on the bootstrap (one way to go) and numerous machine learning ventures (another direction). 
The entire concept of robustness belongs to classical, statistical theory, I believe. Machine learning people are more concerned with stability. It's a very different framework.
