# Is there a Bayesian approach to density estimation

I am interested to estimate the density of a continuous random variable $X$. One way of doing this that I learnt is the use of Kernel Density Estimation.

But now I am interested in a Bayesian approach that along the following lines. I initially believe that $X$ follows a distribution $F$. I take $n$ readings of $X$. Is there some approach to update $F$ based on my new readings?

I know I sound like I am contradicting myself: If I believe solely in $F$ as my prior distribution, then no data should convince me otherwise. However, suppose $F$ were $Unif[0,1]$ and my data points were like $(0.3, 0.5, 0.9, 1.7)$. Seeing $1.7$, I obviously cannot stick to my prior, but how should I update it?

Update: Based on the suggestions in the comments, I have started looking at Dirichlet process. Let me use the following notations:

$G \sim DP(\alpha,H)\\ \theta_i | G \sim G\\ x_i | \theta_i \sim N(\theta_i,\sigma^2)$

After framing my original problem in this language, I guess I am interested in the following: $\theta_{n+1} | x_1,...,x_n$. How does one do this?

In this set of notes (page 2), the author did an example of $\theta_{n+1} | \theta_1,...,\theta_n$ (Polya Urn Scheme). I am not sure if this is relevant.

Update 2: I also wish to ask (after seeing the notes): how do people choose $\alpha$ for the DP? It seems like a random choice. In addition, how do people choose a prior $H$ for DP? Should I just use a prior for $\theta$ as my prior for $H$?

• "If I believe solely in F as my prior distribution, then no data should convince me otherwise." This is the antithesis of Bayesian inference, which is more along the lines of take what you believe in one hand and the world in the other hand, and mush them together and see what come out. Wash, rinse, repeat. Jul 18 '14 at 15:45
• Do you know something about the dirichlet process? Jul 18 '14 at 17:25
• Ignoring your last paragraph: there are two common options to this problem. One is a finite mixture of normals (you can choose how many normals based on likelihood in cross validation) or an infinite mixture of normals as @niandra82 is suggesting. These can be done with something like Gibbs sampling or variational inference.. Are you familiar with any of these methods?
– user44764
Jul 18 '14 at 18:06
• I should also ask, how do you intend to use this KDE? The method chosen and the size (infinite, finite) might depend on your aim.
– user44764
Jul 18 '14 at 18:18
• This sounds like either a model choice problem or a philosophical one. In reality, our choice of which likelihood to use in Bayesian inference imposes prior beliefs too ... Jul 20 '14 at 5:16

Since you want a bayesian approach, you need to assume some prior knowledge about the thing you want to estimate. This will be in the form of a distribution.

Now, there's the issue that this is now a distribution over distributions. However, this is no problem if you assume that the candidate distributions come from some parameterized class of distributions.

For example, if you want to assume the data is gaussian distributed with unknown mean but known variance, then all you need is a prior over the mean.

MAP estimation of the unknown parameter (call it $\theta$) could proceed by assuming that all the observations / data points are conditionally independent given the unknown parameter. Then, the MAP estimate is

$\hat{\theta} = \arg \max_\theta ( \text{Pr}[x_1,x_2,...,x_n,\theta] )$,

where

$\text{Pr}[x_1,x_2,...,x_n,\theta] = \text{Pr}[x_1,x_2,...,x_n | \theta] \text{Pr}[\theta] = \text{Pr}[\theta] \prod_{i=1}^n \text{Pr}[x_i | \theta]$.

It should be noted that there are particular combinations of the prior probability $\text{Pr}[\theta]$ and the candidate distributions $\text{Pr}[x | \theta]$ that give rise to easy (closed form) updates as more data points are received.

For density estimation purposes what you need is not

$\theta_{n+1}|x_{1},\ldots,x_{n}$.

The formula in notes $\theta_{n+1}|\theta_{1},\ldots,\theta_{n}$ reffers to the predictive distribution of the Dirichlet process.

For density estimation you actually have to sample from the predictive distribution $$\pi(dx_{n+1}|x_{1},\ldots,x_{n})$$

Sampling from the above distribution can be done either with conditional methods either with marginal methods. For the conditional methods, take a look at the paper of Stephen Walker [1]. For marginal methods you should check at Radford Neal paper [2].

For the concnetration parameter $\alpha$ Mike West [3] proposes a method for inference in the MCMC procedure including a full conditional distribution for $\alpha$. If you decide not to update the concentration $\alpha$ in the MCMC procedure, you should keep in mind that if you choose a large value for it, then the number of distinct values drawn from the Dirichlet process will be larger than the number of distinct values when a small number for $\alpha$ will be used.

[1] S.G., Walker (2006). Sampling the Dirichlet Mixture model with slices. Communications in Statitics (Simulation and Computation).

[2] R.M., Neal (2000) Markov Chain Monte Carlo methods for Dirichlet Process Mixture models. Journal of Computational and Graphical Statistics. Vol 9, No 2, pp 249-265

[3] M., West (1992). Hyperparameter estimation in Dirichlet process mixture models. Technical report

Is there some approach to update F based on my new readings?

There is something precisely for that. It's pretty much the main idea of Bayesian inference.

$p(\theta | y) \propto p(y|\theta)p(\theta)$

The $p(\theta)$ is your prior, what you call $F$. The $p(y|\theta)$ is what Bayesians call the "likelihood" and it is the probability of observing your data given some value of theta. You just multiply them together and get what's called a "posterior" distribution of $\theta$. This is your "updated F". Check out chapter 1 of any Intro to Bayesian Stats book.

You don't have to get rid of $p(\theta)$ (your prior), you just have to realize that it's not your best guess anymore, now that you have data to refine it.

• This is not answering what the question is asking. OP is asking how one can put a prior on $F$ when $X_1, \ldots, X_n \stackrel{iid}{\sim} F$. Assuming our prior on $F$ puts probability one on distributions with a density, the likelihood is $L(F) = \prod_{i=1}^N \left.\frac{dF}{dx}\right|_{x = x_i}$. So we need to construct a prior on the space of distribution functions $F$ which are differentiable (which is infinite dimensional), and OP is asking how to do this.
– guy
Sep 13 '14 at 21:02