Is there a Bayesian approach to density estimation

Question

I am interested to estimate the density of a continuous random variable $X$. One way of doing this that I learnt is the use of Kernel Density Estimation.

But now I am interested in a Bayesian approach that along the following lines. I initially believe that $X$ follows a distribution $F$. I take $n$ readings of $X$. Is there some approach to update $F$ based on my new readings?

I know I sound like I am contradicting myself: If I believe solely in $F$ as my prior distribution, then no data should convince me otherwise. However, suppose $F$ were $Unif[0,1]$ and my data points were like $(0.3, 0.5, 0.9, 1.7)$. Seeing $1.7$, I obviously cannot stick to my prior, but how should I update it?

Update: Based on the suggestions in the comments, I have started looking at Dirichlet process. Let me use the following notations:

$ G \sim DP(\alpha,H)\\ \theta_i | G \sim G\\ x_i | \theta_i \sim N(\theta_i,\sigma^2)$

After framing my original problem in this language, I guess I am interested in the following: $\theta_{n+1} | x_1,...,x_n$. How does one do this?

In this set of notes (page 2), the author did an example of $\theta_{n+1} | \theta_1,...,\theta_n$ (Polya Urn Scheme). I am not sure if this is relevant.

Update 2: I also wish to ask (after seeing the notes): how do people choose $\alpha$ for the DP? It seems like a random choice. In addition, how do people choose a prior $H$ for DP? Should I just use a prior for $\theta$ as my prior for $H$?

Answer 1

Since you want a bayesian approach, you need to assume some prior knowledge about the thing you want to estimate. This will be in the form of a distribution.

Now, there's the issue that this is now a distribution over distributions. However, this is no problem if you assume that the candidate distributions come from some parameterized class of distributions.

For example, if you want to assume the data is gaussian distributed with unknown mean but known variance, then all you need is a prior over the mean.

MAP estimation of the unknown parameter (call it $\theta$) could proceed by assuming that all the observations / data points are conditionally independent given the unknown parameter. Then, the MAP estimate is

$\hat{\theta} = \arg \max_\theta ( \text{Pr}[x_1,x_2,...,x_n,\theta] )$,

where

$ \text{Pr}[x_1,x_2,...,x_n,\theta] = \text{Pr}[x_1,x_2,...,x_n | \theta] \text{Pr}[\theta] = \text{Pr}[\theta] \prod_{i=1}^n \text{Pr}[x_i | \theta]$.

It should be noted that there are particular combinations of the prior probability $\text{Pr}[\theta]$ and the candidate distributions $\text{Pr}[x | \theta]$ that give rise to easy (closed form) updates as more data points are received.

Answer 2

For density estimation purposes what you need is not

$\theta_{n+1}|x_{1},\ldots,x_{n}$.

The formula in notes $\theta_{n+1}|\theta_{1},\ldots,\theta_{n}$ reffers to the predictive distribution of the Dirichlet process.

For density estimation you actually have to sample from the predictive distribution $$ \pi(dx_{n+1}|x_{1},\ldots,x_{n}) $$

Sampling from the above distribution can be done either with conditional methods either with marginal methods. For the conditional methods, take a look at the paper of Stephen Walker [1]. For marginal methods you should check at Radford Neal paper [2].

For the concnetration parameter $\alpha$ Mike West [3] proposes a method for inference in the MCMC procedure including a full conditional distribution for $\alpha$. If you decide not to update the concentration $\alpha$ in the MCMC procedure, you should keep in mind that if you choose a large value for it, then the number of distinct values drawn from the Dirichlet process will be larger than the number of distinct values when a small number for $\alpha$ will be used.

[1] S.G., Walker (2006). Sampling the Dirichlet Mixture model with slices. Communications in Statitics (Simulation and Computation).

[2] R.M., Neal (2000) Markov Chain Monte Carlo methods for Dirichlet Process Mixture models. Journal of Computational and Graphical Statistics. Vol 9, No 2, pp 249-265

[3] M., West (1992). Hyperparameter estimation in Dirichlet process mixture models. Technical report

Answer 3

Is there some approach to update F based on my new readings?

There is something precisely for that. It's pretty much the main idea of Bayesian inference.

$p(\theta | y) \propto p(y|\theta)p(\theta)$

The $p(\theta)$ is your prior, what you call $F$. The $p(y|\theta)$ is what Bayesians call the "likelihood" and it is the probability of observing your data given some value of theta. You just multiply them together and get what's called a "posterior" distribution of $\theta$. This is your "updated F". Check out chapter 1 of any Intro to Bayesian Stats book.

You don't have to get rid of $p(\theta)$ (your prior), you just have to realize that it's not your best guess anymore, now that you have data to refine it.