5
$\begingroup$

In the Bayes rule, it is said that the posterior $$ P(\theta|D) = \frac{P(D|\theta)P(\theta)}{P(D)} $$ is intractable, because $$ P(D) = \int P(D,\theta) d\theta $$ and the latter is often a high-dimensional integral.

See Why is the posterior distribution in Bayesian Inference often intractable?

But this is just one way of computing $P(D)$. There are others? What about instead estimating $P(D)$ using a kernel density (place a Gaussian or some other lobe at each datapoint, and normalize so it all sums to one). Or simply using delta functions: $$ P(x) = \frac{1}{n} \sum_i \delta_{x_i}(x) $$

This requires touching each bit of data, but that is not intractable.

$\endgroup$
  • 1
    $\begingroup$ What is $x$ in the right hand side of your final equation? The required $P(D)$ is a number, not a function of some $x$ - $\endgroup$ – Juho Kokkala Aug 29 '17 at 17:16
  • 2
    $\begingroup$ A kernel density estimate could be used for estimating a density if we had iid. drawns $x_1,\ldots,x_n$ from some distribution $P(x)$ but that's not at all what is going on here, so I suspect this question is based on a misunderstanding of the basics of Bayesian inference and does not have any useful answers that are not basically introduction to Bayesian inference. But it is possible I am misunderstanding the proposed approach here - if so, could you show how to apply it with some example model? $\endgroup$ – Juho Kokkala Aug 29 '17 at 17:26
  • 1
    $\begingroup$ I am a beginner, so I'm sure it is me who is misunderstanding. The "x" in the last equation was meant to align with P(X) on the left hand side, is a particular arbitrary data value. I think the kernel density approach is widely used for estimating a density, with recognition that it is approximate and does not work well in high dimensions. So I guess my question is, why not use it to compute the posterior. I think there is an obvious reason, I just do not know it. $\endgroup$ – Bull Sep 2 '17 at 11:30
  • 1
    $\begingroup$ I do not know how you would use it to estimate a posterior - a kernel density estimator is used when you have samples from the distribution whose density you are estimating - but this is not the Bayesian inference setting $\endgroup$ – Juho Kokkala Sep 2 '17 at 11:34
  • 2
    $\begingroup$ Let's go through a one-dimensional toy example where the parameter of interest is the mean of a normal distribution: $X_i \sim N(\theta,1)$, conditional on $\theta$ the $X_i$s are independent. The prior is $\theta \sim N(0,1)$. The data is $X_1=1,~X_2=0.5,X_3=0.6$. Now, how do you use the kernel density estimator to compute $P(\theta \mid X_1,X_2,X_3)$? (This case is analytically tractable but it should still be possible to illustrate the method) $\endgroup$ – Juho Kokkala Sep 2 '17 at 11:41
3
$\begingroup$

There are certainly lots of ways to try to numerically estimate high-dimensional definite integrals. The entire field of high-dimensional numerical integration is devoted to this problem, and it suffers from the dreaded curse of dimensionality. There are a lot of research papers in this field with a lot of different methods used. Kernel methods are one method that can be used to obtain approximate integrals (using the delta function would give a terrible approximation for continuous distributions), but I think it is fair to say that the most favoured methods presently used in this field are Monte-Carlo methods (e.g., importance sampling), Markov-Chain Monte-Carlo methods (e.g., Gibbs, Metropolis-Hastings, Hamiltonian MC), and sparse-grid methods.

Most Bayesians make extensive use of Markov-Chain Monte-Carlo (MCMC) methods, and many general pieces of Bayesian software are built on these algorithms. The Stan package for Bayesian statistics is built on using Hamiltonian Monte-Carlo methods to estimate these integrals. This is a powerful method that has led to recent improvements in computational power in Bayesian analysis. I'm not an expert on this stuff myself, but I know it is a very large an complicated field, with lots of methods and lots of literature.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

Since the marginal density writes as$$m(x)=\int_\Theta f(x|\theta),\pi(\theta)\,\text{d}\theta$$a possible numerical approximation is$$\frac{1}{T}\sum_{t=1}^T f(x|\theta_t)\qquad\theta_1,\ldots,\theta_T\sim\pi(\theta)\tag{1}$$but this Monte Carlo approximation based on simulations does not use kernel estimation. As stressed by Juko Kokkala's comments, the use of a kernel estimator of $m(x)$ would require observations from the marginal, while the classical Bayesian framework only involves observations from $f(x|\theta_0)$ for an unknown $\theta_0$. Plus, (1) is a parametric estimator that converges at the rate $\sqrt{T}$, as opposed to a non-parametric estimator that converges at the rate $ T^{−4/2(d+4)}$ where $d$ is the dimension of $x$.

Note also that an intractable posterior is usually understood as associated with an intractable product 'prior x likelihood' rather than having an unknown normalisation constant $m(x)$, since simulation techniques (like MCMC, importance sampling, &tc.) can bypass this missing term and still deliver. (This issue is also discussed in the post the OP linked to.)

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.