# Bayesians: slaves of the likelihood function?

In his book "All of Statistics", Prof. Larry Wasserman presents the following Example (11.10, page 188). Suppose that we have a density $f$ such that $f(x)=c\,g(x)$, where $g$ is a known (nonnegative, integrable) function, and the normalization constant $c>0$ is unknown.

We are interested in those cases where we can't compute $c=1/\int g(x)\,dx$. For example, it may be the case that $f$ is a pdf over a very high-dimensional sample space.

It is well known that there are simulation techniques that allow us to sample from $f$, even though $c$ is unknown. Hence, the puzzle is: How could we estimate $c$ from such a sample?

Prof. Wasserman describes the following Bayesian solution: let $\pi$ be some prior for $c$. The likelihood is $$L_x(c) = \prod_{i=1}^n f(x_i) = \prod_{i=1}^n \left(c\,g(x_i)\right) = c^n \prod_{i=1}^n g(x_i) \propto c^n \, .$$ Therefore, the posterior $$\pi(c\mid x) \propto c^n \pi(c)$$ does not depend on the sample values $x_1,\dots,x_n$. Hence, a Bayesian can't use the information contained in the sample to make inferences about $c$.

Prof. Wasserman points out that "Bayesians are slaves of the likelihood function. When the likelihood goes awry, so will Bayesian inference".

My question for my fellow stackers is: Regarding this particular example, what went wrong (if anything) with Bayesian methodology?

P.S. As Prof. Wasserman kindly explained in his answer, the example is due to Ed George.

-
This example sounds merely like a bizarre ineffective way to conduct numerical integration rather than like any Bayesian analysis. –  whuber Oct 1 '12 at 21:41
How can you say the Bayesian learns nothing about $c$. If this was the case we would have $\pi(c|x)\propto\pi(c)$. It is clearly not. –  probabilityislogic Oct 1 '12 at 22:37
I don't really understand this example. If $g()$ does not depend on $c$ then isn't it unsurprising that the data are not informative as then $c$ depends only on the form of $g()$ and is the same for $any$ sample? I'm obviously missing some subtle (or not so subtle) point. –  Dikran Marsupial Oct 2 '12 at 19:59
I have contrived a formally Bayesian approach that may overcome @Zen's objection, does not contraindicate Xi'an's lack of interest and ends up just assessing the accuracy of numerical integration. –  phaneron Oct 25 '12 at 18:42
A nice follow up on Larry's blog: normaldeviate.wordpress.com/2012/10/05/… –  Zen Apr 24 at 3:42

This has been discussed in my paper (published only on the internet) "On an Example of Larry Wasserman" [1] and in a blog exchange between me, Wasserman, Robins, and some other commenters on Wasserman's blog: [2]

The short answer is that Wasserman (and Robins) generate paradoxes by suggesting that priors in high dimensional spaces "must" have characteristics that imply either that the parameter of interest is known a priori with near certainty or that a clearly relevant problem (selection bias) is known with near certainty not to be present. In fact, sensible priors would not have these characteristics. I'm in the process of writing a summary blog post to draw this together. There is an excellent 2007 paper, showing sensible Bayesian approaches to the examples Wasserman and Ritov consider, by Hameling and Toussaint: “Bayesian estimators for Robins-Ritov’s problem” [3]

-
Thank you for your contribution, Prof. Sims. Do you agree with my answer bellow? P.S. Now we have Nobel Prizes posting on SE. How about that? nobelprize.org/nobel_prizes/economics/laureates/2011/sims.html –  Zen Oct 2 '12 at 2:16
@ChrisSims Professor Sims Thanks for coming on and blowing away my answer with your very authoritative response! –  Michael Chernick Oct 2 '12 at 15:10
I'm alarmed by the fact that this answer has the highest vote total (as of right now). As Prof. Wasserman notes, Prof. Sims's answer is about a completely different puzzle than the one Zen asked about. I infer that most people upvoted it without having read and understood the links Sims provided. –  Cyan Oct 4 '12 at 5:32
Cyan, you can find Prof. Sim's comments regarding this puzzle in Link [1], WassermanComment.pdf, p. 10, Section VII. Postscript 2. –  madprob Oct 8 '12 at 23:18

I agree that the example is weird. I meant it to be more of a puzzle really. (The example is actually due to Ed George.)

It does raise the question of what it means for something to be "known". Christian says that $c$ is known. But, at least from the purely subjective probability point of view, you don't know it just because it can in principle be known. (Suppose you can't do the numerical integral.) A subjective Bayesian regards everything as a random variable with a distribution, including $c$.

At any rate, the paper

A. Kong, P. McCullagh, X.-L. Meng, D. Nicolae, and Z. Tan (2003), A theory of statistical models for Monte Carlo integration, J. Royal Statistic. Soc. B, vol. 65, no. 3, 585–604

(with discussion) treats essentially the same problem.

The example that Chris Sims alludes to in his answer is of a very different nature.

-
Professor Wasserman Thank you for coming on and explaining your example and its history. I was a graduate student at Stanford and overlapped with Ed George. Stanford Statistics Department was very non-Bayesian in those days although with Efron and Stein we were on thee fringe of empirical Bayes. The department was very openminded though and Dennis Lindley gave a graduate course in Bayesian statistics that I took one summer. Somehow Ed got converted to becoming a full-fledged Bayesian and even wrote a paper on Gibbs sampling for dummies (though not with that title of course). –  Michael Chernick Oct 2 '12 at 15:17
I have and enjoy reading your little books "All of Statistics" and "All of Nonparametrics". –  Michael Chernick Oct 2 '12 at 15:18
thanks. glad you enjoyed the books –  Larry Wasserman Oct 2 '12 at 23:53
maybe not so-coincidentally, I discussed this paper by Kong et al. (2003), being mostly negative about the efficiency of using group transformations on the measure rather than on the distribution. Lately, Xiao-Li set me towards a more positive perception of the paper... –  Xi'an Oct 3 '12 at 4:59
"Suppose you can't do the numerical integral." I understand that logical uncertainty (which this is an example of) has resisted analysis despite considerable efforts. –  John Salvatier Oct 3 '12 at 20:52
show 1 more comment

I do not see much appeal in this example, esp. as a potential criticism of Bayesians and likelihood-wallahs.... The constant $c$ is known, being equal to $$1\big/ \int_\mathcal{X} g(x) \text{d}x$$ If $c$ is the only "unknown" in the picture, given a sample $x_1,\ldots,x_n$, then there is no statistical issue about the problem and I do not agree that there exists estimators of $c$. Nor priors on $c$ (other than the Dirac mass on the above value). This is not in the least a statistical problem but rather a numerical issue.

That the sample $x_1,\ldots,x_n$ can be used through a (frequentist) density estimate to provide a numerical approximation of $c$ is a mere curiosity. Not a criticism of alternative statistical approaches: I could also use a Bayesian density estimate...

-
(+1) Agreed. But if we still try to play this artificial game introducing a random normalization constant $C$, I believe that the statistical model is not properly defined. If we can start with a proper prior and get an improper posterior, then I will quit inference. –  Zen Oct 2 '12 at 14:30
It is not possible to start with a proper prior and end up with an improper posterior if the likelihood is a true conditional density! –  Xi'an Oct 2 '12 at 19:55
(+1) Exactly. That's the contradiction that I've presented in my answer. –  Zen Oct 2 '12 at 20:51
I am not de Finetti, so I cannot answer for him! –  Xi'an Oct 3 '12 at 5:00
Your example is statistical: I get observations whose underlying distribution is governed by an unknown parameter c. Larry's (or Ed's!) example is not statistical: the distribution of the sample is completely known and not driven by an unknown parameter c. This is further illustrated by Zen's answer: you simply cannot write $f(x_1,\ldots,x_n|c)$ without ending with a paradox, because there is only one single possible value of c. –  Xi'an Oct 3 '12 at 5:06
show 1 more comment

The proposed statistical model may be described as follows: You have a known nonnegative integrable function $g:\mathbb{R}\to\mathbb{R}$, and a nonnegative random variable $C$. The random variables $X_1,\dots,X_n$ are supposed to be conditionally independent and identically distributed, given that $C=c$, with conditional density $f_{X_i\mid C}(x_i\mid c)=c\,g(x_i)$, for $c>0$.

Unfortunately, in general, this is not a valid description of a statistical model. The problem is that, by definition, $f_{X_i\mid C}(\,\cdot\mid c)$ must be a probability density for almost every possible value of $c$, which is, in general, clearly false. In fact, it is true just for the single value $c=\left(\int_{-\infty}^\infty g(x)\,dx\right)^{-1}$. Therefore, the model is correctly specified only in the trivial case when the distribution of $C$ is concentrated at this particular value. Of course, we are not interested in this case. What we want is the distribution of $C$ to be dominated by Lebesgue measure, having a nice pdf $\pi$.

Hence, defining $x=(x_1,\dots,x_n)$, the expression $$L_x(c) = \prod_{i=1}^n \left(c\,g(x_i)\right) \, ,$$ taken as a function of $c$, for fixed $x$, does not correspond to a genuine likelihood function.

Everything after that inherits from this problem. In particular, the posterior computed with Bayes's Theorem is bogus. It's easy to see that: suppose that you have a proper prior $$\pi(c) = \frac{1}{c^2} \,I_{[1,\infty)}(c) \, .$$ Note that $\int_0^\infty \pi(c)\,dc=1$. According to the computation presented in the example, the posterior should be $$\pi(c\mid x) \propto \frac{1}{c^{2-n}}\, I_{[1,\infty)}(c) \, .$$ But if that is right, this posterior would be always improper, because $$\int_0^\infty \frac{1}{c^{2-n}}\,I_{[1,\infty)}(c)\,dc$$ diverges for every sample size $n\geq 1$.

This is impossible: we know that if we start with a proper prior, our posterior must always be proper. If not, we have just proved that the probability calculus is logically inconsistent.

-
I am sorry that no one has commented on your answer. I think you may have a point but I am a little puzzled. Certainly you can put valid prior distributions on the positive real numbers. Why couldn't you define a probability density f for every c>0 if g is nonegative having a finite integral on R$^+$? –  Michael Chernick Oct 2 '12 at 15:58
Michael: consider the prior density defined by $\pi(c)=(1/c^2)\,I_{[1,\infty)}(c)$. Note that $\int_0^\infty \pi(c)\,dc=1$. Hence, $\pi$ is a proper prior; a genuine pdf for the parameter $c$. According to the computation in the book, the posterior density would be $\pi(c\mid x)\propto (1/c^{2-n})\,I_{[1,\infty)}(c)$. But $\int_0^\infty (1/c^{2-n})\,I_{[1,\infty)}(c) \,dc$ diverges for every sample size $n\geq 1$. Therefore, something must be wrong: you started with a proper prior and computed an improper posterior. Is that clear? –  Zen Oct 2 '12 at 17:35
Any proper prior with no nth moment would also work in your example. I agree this is a useful way to show that something is wrong. My thinking is more that the prior is not based on knowledge of $g(.)$. Because you know $g(.)$ there is only one prior consistent with this information. This is the dirac delta function $p(c|g(.))=\delta(c-\int_{0}^{\infty}g(x)dx)$. To use any other prior is logically incorrect. Its kind of like saying $p(Z|XY)\propto p(Z|X)$ when $Y$ is not independent from $Z$ given $X$ –  probabilityislogic Oct 2 '12 at 21:22
I've flagged this answer to ask that the CW be removed. I think the edits are made in good faith and reflect the development of your thought process as this question (and the other answers) evolved. –  cardinal Oct 5 '12 at 17:31
"Community Wiki". After 10 edits (by the original author), any question or answer gets automatically converted, after which votes continue to accrue, but not the associated reputation. :-) –  cardinal Oct 5 '12 at 17:44

The example is a little weird and contrived. The reason the likelihood goes awry is because g is a known function. The only unknown parameter is c which is not part of the likelihood. Also since g is known the data gives you no information about f. When do you see such a thing in practice? So the posterior is just proportional to the prior and all the information about c is in the prior.

Okay but think about it. Frequentists use maximum likelihood and so the frequentist sometimes rely on the likelihood function also. Well the frequentist can estimate parameters in other ways you may say. But this cooked up problem has only one parameter c and there is no information in the data about c. Since g is known there is no statistical problem related to unknown parameters that can be gleaned out of the data period.

-
Thank you, Michael. Weird situation, insn't? Prof. Wasserman suggests the following way to estimate $c$: take any (frequentist) consistent estimator $\hat{f}$ of the density $f$ (some kind of kernel estimator, for example). Choose an arbitrary point $x$ and note that $\hat{c}=\hat{f}(x)/g(x)$ is a consistent estimator of $c$. –  Zen Oct 1 '12 at 21:46
@Zen Okay let's take that example. Why collect any data at all? We know g. So we can numerically integrate it to determine c to whatever level of accuracy we desire without having to estimate anything! The assumption that we can't compute c which means that even though we know g as a function of x we can't integrate it! I think his example is weak and so is the argument and I like his books generally speaking. –  Michael Chernick Oct 1 '12 at 22:03

There is an irony that the standard way to do Bayesian computation is to use frequentist analysis of MCMC samples. In this example we might consider $c$ to be closely related to the marginal likelihood, which we would like to calculate, but we are going to be Bayesian purists in the sense of to try to also do the computation in a Bayesian way.

It is not common, but it is possible to do this integral in a Bayesian framework. This involves putting a prior on the function $g()$ (in practice a Gaussian process) evaluating the function at some points, conditioning upon these points and computing an integral over the posterior over $g()$. In this situation the likelihood involves evaluating $g()$ at a number of points, but $g()$ is otherwise unknown, therefore the likelihood is quite different to the likelihood given above. The method is demonstrated in this paper http://mlg.eng.cam.ac.uk/zoubin/papers/RasGha03.pdf

I don't think anything went wrong with Bayesian methodology. The likelihood as written treats $g()$ as known everywhere. If this were the case then there would be no statistical aspect to the problem. If $g()$ is assumed to be unknown except at a finite number of points Bayesian methodology works fine.

-
Surprised this doesn't have more upvotes. This gets to the heart of the issue, which is the ambiguous assertion that you "know" what a function is just because you can evaluate it at any point. I think a more appropriate criterion to say you "know" a function is the ability to evaluate any continuous linear functional on it. –  Nick Alger Oct 26 '12 at 11:57
@Nick Alger: Folks have likley lost interest. I am not upvoting it because I am not convinced it is Bayes - do the x.i in the set D (x.i,f(x.i)) refer to x.i observed in the study or randomly generated by them? If it is the first, it is Bayes but very easy to beat with simple MC with a few seconds of computing time (so it does not work fine) or its not Bayes (has not conditioned on the data). –  phaneron Oct 31 '12 at 18:53

We could extend the definition of possible knowns (analogous to the extension of data to allow for missing data for datum that was observed but lost) to include NULL (no data generated).

Suppose that you have a proper prior $$\pi(c) = \frac{1}{c^2} \,I_{[1,\infty)}(c) \, .$$ Now define the data model for x

If $c=\left(\int_{-\infty}^\infty g(x)\,dx\right)^{-1}$

$f_{X_a\mid C}(x_a\mid c) f_{X_i\mid C}(x_i\mid c) =c\,1 g(x_i)$ {a for any}

Otherwise $f_a{X_a\mid C}(x_a\mid c)=0$

So the posterior would be 0 or 1 (proper) but the likelihood from the above data model is not available (because you cannot determine the condition required in the data model.)

So you do ABC.

Draw a “c” from the prior.

Now approximate $\left(\int_{-\infty}^\infty g(x)\,dx\right)^{-1}$ by some numerical integration and keep “c” if that approximation – “c” < epsilon.

The kept “c’s will be an approximation of the true posterior.

(The accuracy of the approximation will depend on epsilon and the sufficiency of conditioning on that approximation.)

-
Wait, what? You have $$\pi(c|x) = \left( \Pi_i g(x_i) \right) \cdot c^n \pi(c) \,,$$ so it does depend on the values of $\{x_i\}$. Just because you hide the dependency in a "$\propto$" doesn't mean you can ignore it?
Dear confused: the equation above is wrong. Where is the denominator (the marginal likelihood of $x$)? Divide by $\int f(x\mid c)\,\pi(c)\,dc$ and you will see that the $\prod_{i=1}^n g(x_i)$ will cancel out. The "posterior" in the book is wrong for other reasons. Please, check my answer. –  Zen Oct 2 '12 at 0:06