As far as I can tell, given observations coming from an unknown distribution $p$ (purported to belong to some class $\mathcal{C}$ of distributions, say e.g. mixtures of Gaussians), there are 3 possible "goals":

  • if $\mathcal{C}$ is parametric: estimate the parameters of $p$ ("parameter estimation")

  • estimate $p$ (say, in $L_2$, KL or $L_1$ distance) by a density $\hat{p}\in\mathcal{C}$ ("proper learning")

  • estimate $p$ (say, in $L_2$, KL or $L_1$ distance) by a density $\hat{p}$ not necessarily in $\mathcal{C}$, e.g. from a superclass $\mathcal{H}\supseteq \mathcal{C}$ ("improper learning")

Intuitively, these tasks range from the hardest to the simplest, and the last two are equivalent as long as computational efficiency is not required.

My question is: what is the motivation for the third one? I.e., in what practical or semi-applied setting would it be beneficial to have in one's hands a good approximation of the distribution with say a kernel estimate, while the original is assumed to be a mixture of 3 Gaussians and an exponential?

To be clear: i understand the theoretical challenges to achieve even the third task, and that in some cases the first two are either too hard or seem hopeless, so tackling (3) is at least a good first step. My question is: in what situations would a practitioner be satisfied with the improper density learning, and what would that practitioner do with the hypothesis $\hat{p}\notin\mathcal{C}$?


Motivations I can think of are:

  • being able to approximately predict the probability of any possible event (and given the type of hypothesis, compute them easily) -- useful e.g. for risk analysis (?)

  • being able to generate more samples roughly distributed as they "should." This could be very useful in experimental sciences: while having few "real" samples from sensors, one can then still be able to run models and simulations requiring many more -- by generating them from the hypothesis (this, of course, requires the said models to be themselves robust).

PS: references or pointers to back up possible applications would be great.


1 Answer 1


If we have a good expectation of the parametric form of the distribution, then it is often the case that we can get a better estimate of the distribution by parameter-fitting than by smoothing. But if we aren't confident of the form of the distribution--i.e. we are worried that we will misspecify the parametric form--then smoothing is often an attractive alternative, particularly in scenarios where it is cheap to collect a very large number of measurements, yielding arbitrarily good empirical nonparametric approximations to the distribution.

One very common case is summarizing the posterior distributions from MCMC output in Bayesian statistical applications. A nonparametric empirical approximation to the distribution is all that is typically desired.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.