I'm thinking about univariate density estimation.

Original Question

In parametric inference, you assume the data are generated from a density that can be summarized by finitely-many parameters. You place a prior on those parameters and try to make inference about them.

In nonparametric inference, you don't make a parametric assumption. You place a prior on the (infinite-dimensional) space of probability density functions and try to make inference about the density function directly.

Question: is parametric inference just a special case of nonparametric inference? That is, by making a parametric assumption and placing a prior on the parameters, are you simply defining a very dogmatic prior on the unrestricted space of densities - one that places no "mass" on densities outside the parametric family?


I think the following is closer to what I had in mind when I asked the original question. Denote the space of probability density functions by $$\mathscr{D}=\left\{f:\mathbb{R}\to[0,\,\infty):\int f(x)dx=1\right\},$$ and denote a generic parametric family by $$\mathscr{P}=\{f_\theta\in\mathscr{D}:\theta\in\Theta\subseteq\mathbb{R}^p\}.$$ If you place a probability measure $P$ on $(\Theta,\,\mathcal{B})$, I think it induces a probability measure on the infinite-dimensional function space $\mathscr{P}$ as follows:

  • For each Borel set $E\in\mathcal{B}$, define $$F(E):=\{f_\theta\in\mathscr{P}:\theta\in E\};$$

  • You can show that $\mathcal{F}(\mathcal{B}):=\{F(E):E\in\mathcal{B}\}$ is a $\sigma$-algebra on $\mathscr{P}$;

  • Set $P(f_\theta\in F(E)):=P(\theta\in E)$ for each $F(E)\in\mathcal{F}(\mathcal{B})$.

Question: In what sense does the probability measure $P$ on $(\Theta,\,\mathcal{B})$ induce a probability measure on all of $\mathscr{D}$?

  • 1
    $\begingroup$ In essence, also having exactly one distribution in the prior is a special case of nonparametric Bayesian inference. However, you can see that the resulting estimates behave differently on the samples (with one distribution prior - the posterior distribution is fixed). What aspect of Bayesian estimation are you looking to compare? $\endgroup$
    – tmrlvi
    Jul 29, 2018 at 20:38
  • $\begingroup$ @tmrlvi Thanks for your response. How far can I take the logic in practice? Say I pick a parametric family and its conjugate prior such that the posterior is known. Can I write the parametric assumption + conjugate prior as a prior on the space of densities, use some numerical method to approximate the posterior over the space of densities, and then see that it "looks like" the conjugate posterior? $\endgroup$
    – jcz
    Jul 30, 2018 at 4:02

1 Answer 1


The answer to your question is "yes" within the context of the particular nonparametric framework that I describe below.

Bayesian density "estimation" can understood as computing a predictive distribution for a "future" observation $y_{n+1}$ based on a collection of observations $y_{1:n} = (y_1,\ldots,y_n)$. Suppose the observations are conditionally independent: \begin{equation} p(y_{1:n+1}|\psi) = \prod_{i=1}^{n+1} p(y_i|\psi) \end{equation} for some $p(y_i|\psi)$ where $\psi$ is a parameter (vector). The predictive distribution can be expressed as follows: \begin{equation}\tag{1} p(y_{n+1}|y_{1:n}) = \int p(y_{n+1}|\psi)\,p(\psi|y_{1:n})\,d\psi , \end{equation} where $p(\psi|y_{1:n})$ is the posterior distribution for $\psi$. Equation (1) applies to both parametric and nonparametric models. The form of $p(y_i|\psi)$ determines the type of model.

I'll begin with a nonparametric model and then show how it specializes to a parametric model. The nonparametric approach presented here involves an infinite mixture. As a preliminary, let $\psi = (w,\theta)$, where $w = (w_1, w_2, \ldots)$ is an infinite collection of nonnegative weights that sum to one and $\theta = (\theta_1, \theta_2, \ldots)$ is a corresponding infinite collection of component parameters. Now let \begin{equation} p(y_i|\psi) = \sum_{c=1}^\infty w_c \, f(y_i|\theta_c) , \end{equation} where $f(y_i|\theta_c)$ is a parametric distribution (sometimes called the kernel in this context) where \begin{equation} \int f(y_i|\theta_c)\,dy_i = 1 . \end{equation} The prior for $\psi$ is given by \begin{align} w|\alpha &\sim \textsf{Stick}(\alpha) \\ \theta_c &\stackrel{\text{iid}}{\sim} H , \end{align} where $\alpha$ is called the concentration parameter and $H$ is called the base distribution. The stick-breaking distribution for the weights is characterized by \begin{equation} w_c = v_c \prod_{s=1}^{c-1} (1-v_s) , \qquad\text{where $v_c \stackrel{\text{iid}}{\sim} \textsf{Beta}(1,\alpha)$} . \end{equation}

In order to implement the model, one must specify the kernel and the base distribution. It is convenient --- but not necessary --- to make $H$ be the conjugate prior for $f$. For example, we could have \begin{equation} f(y_i|\theta_c) = \textsf{N}(y_i|\mu_c,\sigma_c^2) , \end{equation} where $\theta_c = (\mu_c,\sigma_c^2)$. In this case, $H$ could be the conjugate normal-inverse-gamma distribution for $(\mu_c,\sigma_c^2)$.

A parametric can be seen as a special limiting case. Note that $E[w_1|\alpha] = (1+\alpha)^{-1}$. Therefore, in the limit as $\alpha \to 0$ the distribution for $w_1$ collapses to a point mass at one and the model reduces to a parametric model: \begin{equation} \lim_{\alpha\to 1}\ p(y_i|\psi) = f(y_i|\theta_1) , \end{equation} where $\theta_1 \sim H$.

As it stands, the nonparametric model is a Dirichlet Process Mixture (DPM) model. It will prove useful to generalize it slightly by putting a prior on the concentration parameter so that \begin{equation} p(\psi,\alpha) = p(\theta)\,p(w|\alpha)\,p(\alpha) . \end{equation} This will allow the data to help determine the value for $\alpha$. In addition, we will obtain the posterior distribution: \begin{equation} p(\psi,\alpha|y_{1:n}) \end{equation} The predictive distribution $p(y_{n+1}|y_{1:n})$ depends only on the marginal posterior $p(\psi|y_{1:n})$.

We can use the marginal posterior for the concentration parameter $p(\alpha|y_{1:n})$ to help make a comparison between the parametric model (the restricted model) and the nonparametric model (the unrestricted model). Let $M_0$ denote the model subject to the restriction $\alpha = 0$ and let $M_1$ denote the unrestricted model. ($M_0$ and $M_1$ are just model labels and do not signify "null" and "alternative" models or hypotheses.)

Bayes rule says \begin{equation} p(\alpha|y_{1:n}) = \frac{p(y_{1:n}|\alpha)\,p(\alpha)}{p(y_{1:n})} , \end{equation} where \begin{equation} p(y_{1:n}|\alpha) = \int p(y_{1:n}|\psi)\,p(\psi,\alpha)\,d\psi \end{equation} and \begin{equation} p(y_{1:n}) = \int p(y_{1:n}|\alpha)\,p(\alpha)\,d\alpha . \end{equation} Rearranging Bayes rule produces \begin{equation}\tag{2} \frac{p(\alpha|y_{1:n})}{p(\alpha)} = \frac{p(y_{1:n}|\alpha)}{p(y_{1:n})} . \end{equation} Equation (2) holds for all values of $\alpha$, including $\alpha = 0$ (at least in a limiting sense). Therefore we have \begin{equation}\tag{3} \frac{p(\alpha=0|y_{1:n})}{p(\alpha=0)} = \frac{p(y_{1:n}|\alpha=0)}{p(y_{1:n})} . \end{equation} The left-hand side of (3) is called the Savage--Dickey density ratio. It is the ratio of the posterior density for $\alpha$ evaluated at $\alpha = 0$ to the prior density evaluated at the same place. The right-hand side of (3) is the Bayes factor in favor of the restricted model relative to the unrestricted model. In other words, by examining what happens to the density of $\alpha$ at zero, we can see whether and how much the data favor the restriction.

You should be aware that many Bayesian practitioners frown on model comparison of this sort, which amounts to a sharpe hypothesis test (because $\alpha = 0$ occupies a set of measure zero in $[0,\infty)$). They correctly note that the Bayes factor in favor of the restriction can be extremely sensitive to the prior for the concentration parameter, $p(\alpha)$. This sensitivity may increase with the sample size $n$. The reason is that the posterior density near $\alpha = 0$ may become less sensitive to the prior as the sample size increases. The upshot is that one should be thoughtful about the comparison. [BTW, unless I have a reason to deviate, my preferred prior for $\alpha$ is $p(\alpha) = (1+\alpha)^{-2}$. The median of this distribution is one and the mean is infinite.]

As a final comment, let me note that I have not touched on how to estimate the DPM. There are a number of ways to do this. My preferred approach is to convert the infinite sum into a finite sum where $m$ is the upper bound ($m$ should be large enough so that there are alway a number of "empty" components). The prior for $w$ then using the truncated stick-breaking prior, which sets $v_m = 1$. In addition, as is typical with mixture models, it is convenient to introduce latent classification variables, $z_{1:n}$, where $z_i = c$ indicates observation $i$ is classified with cluster $c$. In this form the model is quite standard. See for example Bayesian Data Analysis (3rd edition) by Gelman et al.

  • $\begingroup$ Thanks for the thorough response. I learned a lot from studying it. In the framework you describe, it makes sense that the nonparametric case nests the parametric case because you are assuming that the likelihood is an infinite mixture of densities from a particular parametric family. Now that I’ve thought about it more, I think my question has more to do with probability theory than it does with the particulars of Bayesian nonparametric inference per se. I'll update the question a bit. $\endgroup$
    – jcz
    Aug 2, 2018 at 22:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.