# Intractable posterior - why not use kernel density for the data distribution?

In the Bayes rule, it is said that the posterior $$P(\theta|D) = \frac{P(D|\theta)P(\theta)}{P(D)}$$ is intractable, because $$P(D) = \int P(D,\theta) d\theta$$ and the latter is often a high-dimensional integral.

But this is just one way of computing $P(D)$. There are others? What about instead estimating $P(D)$ using a kernel density (place a Gaussian or some other lobe at each datapoint, and normalize so it all sums to one). Or simply using delta functions: $$P(x) = \frac{1}{n} \sum_i \delta_{x_i}(x)$$

This requires touching each bit of data, but that is not intractable.

• What is $x$ in the right hand side of your final equation? The required $P(D)$ is a number, not a function of some $x$ - – Juho Kokkala Aug 29 '17 at 17:16
• A kernel density estimate could be used for estimating a density if we had iid. drawns $x_1,\ldots,x_n$ from some distribution $P(x)$ but that's not at all what is going on here, so I suspect this question is based on a misunderstanding of the basics of Bayesian inference and does not have any useful answers that are not basically introduction to Bayesian inference. But it is possible I am misunderstanding the proposed approach here - if so, could you show how to apply it with some example model? – Juho Kokkala Aug 29 '17 at 17:26
• I am a beginner, so I'm sure it is me who is misunderstanding. The "x" in the last equation was meant to align with P(X) on the left hand side, is a particular arbitrary data value. I think the kernel density approach is widely used for estimating a density, with recognition that it is approximate and does not work well in high dimensions. So I guess my question is, why not use it to compute the posterior. I think there is an obvious reason, I just do not know it. – Bull Sep 2 '17 at 11:30
• I do not know how you would use it to estimate a posterior - a kernel density estimator is used when you have samples from the distribution whose density you are estimating - but this is not the Bayesian inference setting – Juho Kokkala Sep 2 '17 at 11:34
• Let's go through a one-dimensional toy example where the parameter of interest is the mean of a normal distribution: $X_i \sim N(\theta,1)$, conditional on $\theta$ the $X_i$s are independent. The prior is $\theta \sim N(0,1)$. The data is $X_1=1,~X_2=0.5,X_3=0.6$. Now, how do you use the kernel density estimator to compute $P(\theta \mid X_1,X_2,X_3)$? (This case is analytically tractable but it should still be possible to illustrate the method) – Juho Kokkala Sep 2 '17 at 11:41

Most Bayesians make extensive use of Markov-Chain Monte-Carlo (MCMC) methods, and many general pieces of Bayesian software are built on these algorithms. The Stan package for Bayesian statistics is built on using Hamiltonian Monte-Carlo methods to estimate these integrals. This is a powerful method that has led to recent improvements in computational power in Bayesian analysis. I'm not an expert on this stuff myself, but I know it is a very large an complicated field, with lots of methods and lots of literature.
Since the marginal density writes as$$m(x)=\int_\Theta f(x|\theta),\pi(\theta)\,\text{d}\theta$$a possible numerical approximation is$$\frac{1}{T}\sum_{t=1}^T f(x|\theta_t)\qquad\theta_1,\ldots,\theta_T\sim\pi(\theta)\tag{1}$$but this Monte Carlo approximation based on simulations does not use kernel estimation. As stressed by Juko Kokkala's comments, the use of a kernel estimator of $$m(x)$$ would require observations from the marginal, while the classical Bayesian framework only involves observations from $$f(x|\theta_0)$$ for an unknown $$\theta_0$$. Plus, (1) is a parametric estimator that converges at the rate $$\sqrt{T}$$, as opposed to a non-parametric estimator that converges at the rate $$T^{−4/2(d+4)}$$ where $$d$$ is the dimension of $$x$$.
Note also that an intractable posterior is usually understood as associated with an intractable product 'prior x likelihood' rather than having an unknown normalisation constant $$m(x)$$, since simulation techniques (like MCMC, importance sampling, &tc.) can bypass this missing term and still deliver. (This issue is also discussed in the post the OP linked to.)