Are bayesians slaves of the likelihood function?

Question

In his book "All of Statistics", Prof. Larry Wasserman presents the following Example (11.10, page 188). Suppose that we have a density $f$ such that $f(x)=c\,g(x)$, where $g$ is a known (nonnegative, integrable) function, and the normalization constant $c>0$ is unknown.

We are interested in those cases where we can't compute $c=1/\int g(x)\,dx$. For example, it may be the case that $f$ is a pdf over a very high-dimensional sample space.

It is well known that there are simulation techniques that allow us to sample from $f$, even though $c$ is unknown. Hence, the puzzle is: How could we estimate $c$ from such a sample?

Prof. Wasserman describes the following Bayesian solution: let $\pi$ be some prior for $c$. The likelihood is $$ L_x(c) = \prod_{i=1}^n f(x_i) = \prod_{i=1}^n \left(c\,g(x_i)\right) = c^n \prod_{i=1}^n g(x_i) \propto c^n \, . $$ Therefore, the posterior $$ \pi(c\mid x) \propto c^n \pi(c) $$ does not depend on the sample values $x_1,\dots,x_n$. Hence, a Bayesian can't use the information contained in the sample to make inferences about $c$.

Prof. Wasserman points out that "Bayesians are slaves of the likelihood function. When the likelihood goes awry, so will Bayesian inference".

My question for my fellow stackers is: Regarding this particular example, what went wrong (if anything) with Bayesian methodology?

P.S. As Prof. Wasserman kindly explained in his answer, the example is due to Ed George.

Answer 1

I do not see much appeal in this example, esp. as a potential criticism of Bayesians and likelihood-wallahs.... The constant $c$ is known, being equal to $$ 1\big/ \int_\mathcal{X} g(x) \text{d}x $$ If $c$ is the only "unknown" in the picture, given a sample $x_1,\ldots,x_n$, then there is no statistical issue about the problem and I do not agree that there exist estimators of $c$. Nor priors on $c$ (other than the Dirac mass on the above value). This is not in the least a statistical problem but rather a numerical issue.

That the sample $x_1,\ldots,x_n$ can be used through a (frequentist) density estimate to provide a numerical approximation of $c$ is a mere curiosity. Not a criticism of alternative statistical approaches: I could also use a Bayesian density estimate...

Answer 2

This has been discussed in my paper (published only on the internet) "On an Example of Larry Wasserman" [1] and in a blog exchange between me, Wasserman, Robins, and some other commenters on Wasserman's blog: [2]

The short answer is that Wasserman (and Robins) generate paradoxes by suggesting that priors in high dimensional spaces "must" have characteristics that imply either that the parameter of interest is known a priori with near certainty or that a clearly relevant problem (selection bias) is known with near certainty not to be present. In fact, sensible priors would not have these characteristics. I'm in the process of writing a summary blog post to draw this together. There is an excellent 2007 paper, showing sensible Bayesian approaches to the examples Wasserman and Ritov consider, by Hameling and Toussaint: “Bayesian estimators for Robins-Ritov’s problem” [3]

Answer 3

I agree that the example is weird. I meant it to be more of a puzzle really. (The example is actually due to Ed George.)

It does raise the question of what it means for something to be "known". Christian says that $c$ is known. But, at least from the purely subjective probability point of view, you don't know it just because it can in principle be known. (Suppose you can't do the numerical integral.) A subjective Bayesian regards everything as a random variable with a distribution, including $c$.

At any rate, the paper

A. Kong, P. McCullagh, X.-L. Meng, D. Nicolae, and Z. Tan (2003), A theory of statistical models for Monte Carlo integration, J. Royal Statistic. Soc. B, vol. 65, no. 3, 585–604

(with discussion) treats essentially the same problem.

The example that Chris Sims alludes to in his answer is of a very different nature.

Answer 4

The proposed statistical model may be described as follows: You have a known nonnegative integrable function $g:\mathbb{R}\to\mathbb{R}$, and a nonnegative random variable $C$. The random variables $X_1,\dots,X_n$ are supposed to be conditionally independent and identically distributed, given that $C=c$, with conditional density $f_{X_i\mid C}(x_i\mid c)=c\,g(x_i)$, for $c>0$.

Unfortunately, in general, this is not a valid description of a statistical model. The problem is that, by definition, $f_{X_i\mid C}(\,\cdot\mid c)$ must be a probability density for almost every possible value of $c$, which is, in general, clearly false. In fact, it is true just for the single value $c=\left(\int_{-\infty}^\infty g(x)\,dx\right)^{-1}$. Therefore, the model is correctly specified only in the trivial case when the distribution of $C$ is concentrated at this particular value. Of course, we are not interested in this case. What we want is the distribution of $C$ to be dominated by Lebesgue measure, having a nice pdf $\pi$.

Hence, defining $x=(x_1,\dots,x_n)$, the expression $$ L_x(c) = \prod_{i=1}^n \left(c\,g(x_i)\right) \, , $$ taken as a function of $c$, for fixed $x$, does not correspond to a genuine likelihood function.

Everything after that inherits from this problem. In particular, the posterior computed with Bayes's Theorem is bogus. It's easy to see that: suppose that you have a proper prior $$ \pi(c) = \frac{1}{c^2} \,I_{[1,\infty)}(c) \, . $$ Note that $\int_0^\infty \pi(c)\,dc=1$. According to the computation presented in the example, the posterior should be $$ \pi(c\mid x) \propto \frac{1}{c^{2-n}}\, I_{[1,\infty)}(c) \, . $$ But if that is right, this posterior would be always improper, because $$ \int_0^\infty \frac{1}{c^{2-n}}\,I_{[1,\infty)}(c)\,dc $$ diverges for every sample size $n\geq 1$.

This is impossible: we know that if we start with a proper prior, our posterior can't be improper for every possible sample (it may be improper inside a set of null prior predictive probability).

Answer 5

There is an irony that the standard way to do Bayesian computation is to use frequentist analysis of MCMC samples. In this example we might consider $c$ to be closely related to the marginal likelihood, which we would like to calculate, but we are going to be Bayesian purists in the sense of to try to also do the computation in a Bayesian way.

It is not common, but it is possible to do this integral in a Bayesian framework. This involves putting a prior on the function $g()$ (in practice a Gaussian process) evaluating the function at some points, conditioning upon these points and computing an integral over the posterior over $g()$. In this situation the likelihood involves evaluating $g()$ at a number of points, but $g()$ is otherwise unknown, therefore the likelihood is quite different to the likelihood given above. The method is demonstrated in this paper http://mlg.eng.cam.ac.uk/zoubin/papers/RasGha03.pdf

I don't think anything went wrong with Bayesian methodology. The likelihood as written treats $g()$ as known everywhere. If this were the case then there would be no statistical aspect to the problem. If $g()$ is assumed to be unknown except at a finite number of points Bayesian methodology works fine.

Answer 6

The example is a little weird and contrived. The reason the likelihood goes awry is because g is a known function. The only unknown parameter is c which is not part of the likelihood. Also since g is known the data gives you no information about f. When do you see such a thing in practice? So the posterior is just proportional to the prior and all the information about c is in the prior.

Okay but think about it. Frequentists use maximum likelihood and so the frequentist sometimes rely on the likelihood function also. Well the frequentist can estimate parameters in other ways you may say. But this cooked up problem has only one parameter c and there is no information in the data about c. Since g is known there is no statistical problem related to unknown parameters that can be gleaned out of the data period.

Answer 7

We could extend the definition of possible knowns (analogous to the extension of data to allow for missing data for datum that was observed but lost) to include NULL (no data generated).

Suppose that you have a proper prior $$ \pi(c) = \frac{1}{c^2} \,I_{[1,\infty)}(c) \, . $$ Now define the data model for x

If $c=\left(\int_{-\infty}^\infty g(x)\,dx\right)^{-1}$

$f_{X_a\mid C}(x_a\mid c) f_{X_i\mid C}(x_i\mid c) =c\,1 g(x_i)$ {a for any}

Otherwise $f_a{X_a\mid C}(x_a\mid c)=0$

So the posterior would be 0 or 1 (proper) but the likelihood from the above data model is not available (because you cannot determine the condition required in the data model.)

So you do ABC.

Draw a “c” from the prior.

Now approximate $\left(\int_{-\infty}^\infty g(x)\,dx\right)^{-1}$ by some numerical integration and keep “c” if that approximation – “c” < epsilon.

The kept “c’s will be an approximation of the true posterior.

(The accuracy of the approximation will depend on epsilon and the sufficiency of conditioning on that approximation.)

Answer 8

Wait, what? You have $$\pi(c|x) = \left( \Pi_i g(x_i) \right) \cdot c^n \pi(c) \,,$$ so it does depend on the values of $\{x_i\}$. Just because you hide the dependency in a "$\propto$" doesn't mean you can ignore it?