What is the Wine/Water Paradox in Bayesian statistics, and what is its resolution?

Question

I have just heard about the Wine/Water Paradox in Bayesian statistics, but didn't understand it very well (see Mikkelson 2004 for an introduction). Can you explain in simple terms what the paradox is (and why is it a paradox), why it matters for Bayesian statistics, and its resolution?

Answer 1

What the paradox is

There is a mixture of wine and water. Let $x$ be the amount of wine divided by the amount of water. Suppose we know that $x$ is between $1/3$ and $3$ but nothing else about $x$. We want the probability that $x \le 2$.

Without a sample space or probability model, we have no way to calculate probabilities. So we have to decide how to model the problem.

The Principle of Indifference states that if we have no reason to favour one outcome over another, then we should assign them the same probability. This means that we should say that every possible value of $x$ is equally likely. Therefore, the probability that $x \le 2$ is $(2 - 1/3)/(3 -1/3) = 5/8$.

(If you are not comfortable with continuous probability, we could do another version in which $x$ can only take on the values $1/3, 2/3, 1, 4/3, 5/3, 2, 7/3, 8/3, 3$. Then the probability would be $6/9$. This version will lead to the same paradox.)

That's fine, the answer is $5/8$. But now, what would happen if we decided to use the same model, but for the ratio of water divided by wine? Call this $y$. Then $y = 1/x$. Now, if we assume that all values of $y$ are equally likely, we want the probability that $y \ge 1/2$. But this is $(3 - 1/2)/(3-1/3) = 15/16$, (or $8/9$ in the discrete version.)

The paradox is that that these two values are not equal. So, how should we assign the probability that $x \le 2$? Should it be $5/8$ or $15/16$? It depends on our model. But why would we favour one model over the other?

The Principle of Indifference tells us to choose either model, but they give different answers depending on which liquid is called "water" and which liquid is called "wine".

Why it matters for Bayesian statistics

In Bayesian statistics, every calculation is based on choosing a prior distribution for the parameters of interest. For example, if we wanted to make some inference about the wine/water problem, we would have to decide a prior distribution on the ratio of wine and water. Often we want to choose the prior distribution which implies "no prior knowledge", which is usually a uniform or flat prior, which assumes all values are equally likely.

But we have just seen that when we look at things in a different way, "all values of $x$ are equally likely" becomes "all values of $1/x$ are very much not equally likely", so it seems that there is no way to assign a prior distribution of "no information about the value of $x$".

This is rather alarming, since all our calculations will depend on assumptions which we didn't intend to make.

Resolution of the paradox

The paradox has been touted (for over a century) as a refutation of the Principle of Indifference.

Statisticians are happy to say that the Principle isn't valid, and this may be true, but if we can't use the Principle of Indifference, then we can't actually take random samples from anything at all, because even in a computer, sampling is ultimately based on counting the number of possible outcomes among equally likely outcomes.

So what is wrong with the paradox?

The key here is that we do have some prior knowledge about the ratio $x$ of wine to water. Namely, that it is the ratio of wine to water.

In other words, if $z$ is the proportion of water in the mixture, then $x = z/(1-z)$. So saying that all values of $x$ are equally likely is the same as saying that all values of $z/(1-z)$ are equally likely, which seems like an odd thing to assume.

If instead, we assume that all values of $z$ are equally likely, then we get the answer $5/6$, and the paradox vanishes. This is what Mikkelson is getting at in his paper.

Assuming that all values of $z$ are equally likely is a bit like saying "every molecule in the mixture is equally likely to be wine or water, and we are indifferent as to which it is" which seems like a reasonable assumption for this particular situation.

Alternatively, we could view the situation as putting a prior on $x$ proportional to $1/(1+x)^2$. This is called the Jeffreys Prior. Jeffreys was a physicist who had the idea that priors ought to be chosen in such a way as to be invariant to reparametrisations like this.

So he would have said that, if we know the quantity $x$ is a ratio, it's natural to choose this prior instead of any other one.

I am not claiming that I have a resolution of the paradox, or that it's not important. We should definitely be careful about what priors we use and which assumptions we are implicity making. I'm just saying that choosing a prior is more or less the same as choosing a statistical model for something, and we should be careful about choosing these too.

It's a bit unfair to Bayesians to say: "Your choice of prior inevitably leads to a contradiction, but I can choose to model some quantity with a normal distribution or whatever, and it's fine because I can't be bothered to think about these issues."

Notes

Information Geometry

It would be nice if statistics could be made "coordinate-free" so that it doesn't depend on parametrisations. I believe the subject that attempts to do this is called Information Geometry, and it hasn't been found to be of much practical value so far, but you never know.

The Gibbs Paradox

The Principle of Indifference is fundamental to statistical mechanics, which is the branch of physics which describes the behaviour of gases and things. In statistical mechanics, we assume that each possible configuration of particles is equally likely; this is a fundamental assumption which underpins all calculations. This is relevant to the above for two reasons.

In the wine/water problem, statistical mechanics would say that the answer is $5/6$. A physicist would find it very weird to say something like "Let's assume that every possible ratio of hydrogen to oxygen in this container is equally likely."

The second reason is that a paradox involving the Principle of Indifference actually happened in statistical mechanics. It had to be resolved by assuming that particles are indistinguishable, otherwise the theory fails to agree with practical experiments. I am not sure of the details, but you can read up on it under the search term "Gibbs Paradox". The indistinguishability assumption was not theoretically justified until quantum mechanics was developed.

Answer 2

In the context of modern understandings of Bayesian analysis, it is really quite generous to still call this a "paradox". It is nothing more than a demonstration that uniform distributions are not invariant to nonlinear reparameterisations of their referents, such that you have to be careful when forming a "non-informative" prior on an unknown parameter in Bayesian statistics. This was important in the early days of Bayesian statistics, in dealing with some of the first attempts to formulate rules for prior ignorance; nowadays it just illustrates principles that are well-known.

As explained here, the "paradox" arises when we have a wine-water mixture, with unknown composition, and we try to formulate a prior for the ratio of wine-to-water. Suppose we let $x$ be the ratio of wine-to-water and suppose we give this unknown value a uniform prior over its possible range. The so-called "paradox" shows that you get different inferences if you apply a uniform prior to either $x$ or $1/x$ in the problem, despite the fact that there is a clear symmetry to your ignorance about these quantities (the latter is the water-to-wine ratio). This is contrary to some early crude versions of the "principle of indifference" which asserted that ignorance of an unknown quantity should be represented by a uniform prior over the possible values of that quantity.

This "paradox" is trivially resolved in modern Bayesian analysis by recognising that a more natural "non-informative" prior here is uniform over the quantity $z = \tfrac{x}{1+x}$, which is the proportion of wine (or water) in the mixture. Consequently, the problem is just a demonstration that you need to be careful when forming "non-informative" priors, to make sure that they are invariant with respect to natural transforms in the problem. Bayesian literature on non-informative priors is replete with discussions of invariance conditions, so these are issues that are now well-known.$^\dagger$

---

$^\dagger$ The comments seek some additional detail/references to learn about this area of the field. José Bernardo is probaby the leading expert in this field, and so his papers/books are a useful starting point. You can find a useful introduction to the subject in Irony and Singpurwalla (1997) (discussion with José Bernardo).

Answer 3

I believe it to be an apparent paradox and highly instructive of a common and dangerous issue in all branches of statistics, how to handle ratios. Being conscious of a possible paradox will make you cautious as a researcher. I will give you a reason to believe that it is not a paradox.

The problem has no solution, of course. Imposing the principle of indifference doesn’t solve the problem in either case. Where did that choice come from? Why that principle? Using the principle of indifference is itself a subjective choice. There is an infinite number of solutions.

Also, the wording of the problem makes it difficult to work on. So, first, I will give the language of the proposed paradox as it appears in Wikipedia. Then I will unpack it.

A mixture is known to contain a mix of wine and water in proportions such that the amount of wine divided by the amount of water is a ratio $x$ lying in the interval $1/3\leq x\leq 3$. (i.e. 25-75% alcohol) We seek the probability, $P^*$ say, that $x\leq 2$. (i.e., less than or equal to 66%.)

Note that if you were to divide the water by the wine instead and denote this $y$, then $$y=\frac{1}{x}.$$

Note that $$x\leq{2}$$ is equivalent to $$y\ge\frac{1}{2}.$$ The boundaries are still $1/3$ and $3$ but the meaning has changed.

The source of the problem is that the percentage of the total area from 2 to 3 is not the same percentage of the total area from 1/3 to1/2 if both are treated as rectangles of unit size.

The crux of the paradox is that neither frame of reference makes more sense than the other frame of reference. Why should water to wine be the canonical solution to the problem instead of wine to water?

Ratio problems show up all over the place in nature. This is an important warning.

So, now, let us unpack the problem a bit.

First, notice that you have collected no data at all. Although probabilities based only on prior distributions are a significant element of decision making, we tend to ignore them in the pedagogy because they are not computationally intense. Also, as the definition of a statistic is a function of data, then ignoring data isn’t very useful in the field of statistics as a discussion point.

Nonetheless, I am sure you drive many places on the assumption that a meteor has not struck and destroyed the route without collecting any data on local meteor strikes before leaving home.

Your priors over routes are likely not uniform in most places of travel in the United States. Some intersections have long lights; others tend to get congested.

If you had any prior experience of this type of substance, it is likely your prior would not have been uniform, and the principle of indifference would not apply. Any information at all would automatically resolve this paradox because the prior would have to conform to your frame of reference, and the expected probabilities would become equal automatically.

A key element of this paradox is truly having no information.

A possibly missed element of the problem is that you are calculating $P^*$, which is defined here as an expectation. That implies that your loss function is quadratic. Do you have quadratic loss? While the outcome would likely be a paradox under most other loss functions, using a loss function is imposing a Frequentist method on a Bayesian problem.

There is nothing specifically wrong with imposing a Frequentist criterion on a Bayesian problem. That is what Bayesian decision theory is all about. Nonetheless, reducing a distribution down to a point can produce unexpected results. One of the obvious warnings is that Bayesian methods are not invariant to transformations.

Now let us consider an alternate solution.

Let $a=\text{quantity of water}$.

Let $b=\text{quantity of wine}$.

Let $k$ equal the total volume. Assume $k$ is known with certainty.

Instead of solving $$x=\frac{b}{a}$$ we could solve $a+b=k$. Instead of one parameter to estimate, we have two. Under the principle of indifference, $\Pr(a)=2/k,$ when $.25k\le{a}\le{.75}k$ and zero elsewhere. By symmetry, the same prior holds for $b$. Each combination is equiprobable.

Now, your expected ratio is approximately 1.197 for wine to water. There is still a paradox because the expected ratio is also 1.197 for water to wine.

Is that a solution?

Consider an alternative loss function. Consider the loss function $$\mathcal{L}(\theta,\hat{\theta})=|\hat{\theta}-\theta|.$$

In that case, in both ratios, $P^*=.5$, so the ratio is 1.

These are not solutions. Now you have four possible solutions. The final one is attractive, though, because the answer does not depend on the frame of reference.

Another, maybe simpler solution, is to ignore the need for a point estimate of the probability. Just show the one thing that you know, that quantity sits inside a bounded range. The true Bayesian solution would do nothing more than describe your prior distribution. No one point would be favored over others. The decision-theoretic choice of a point is rational if you have a utility function, but is it necessary?

Answer 4

As @Ben's answer points out, such "paradoxes" are better seen as exposing constraints that need to be applied to the prior distribution. The constraints arise from your indifference to particular choices among parametrizations, owing to your ignorance; if you're able to uncover enough "paradoxes", you'll be able to specify a unique prior distribution.

With insufficient reason to treat the wine to water ratio differently from the water to wine ratio (it oughtn't to matter if they're given as 'Liquid A' & 'Liquid B')

$$\Pr(c_1 < X < x) = \Pr\left(\tfrac{1}{x} < \tfrac{1}{X} < \tfrac{1}{c_1}\right)$$

(where $c_1$ is the lower bound on the wine to water ratio), so the prior density functions of $\frac{1}{x}$ & $x$ must be equal,

$$f_{\frac{1}{X}}\left(\tfrac{1}{x}\right) = f_X(x)$$

constraining the allowable prior density functions of $x$ by

$$f_X\left(\tfrac{1}{x}\right) = x^2\cdot f_X(x)$$

Note the proposed solution, $f_X(x)\propto\frac{1}{(1+x)^2}$, i.e. a uniform prior on $z=\frac{x}{1+x}$, does not uniquely satisfy this constraint—& uniform priors on $\operatorname{logit}(z)$ or $\operatorname{probit}(z)$ might seem as natural to some statisticians. You need another "paradox" ...

Suppose the problem were specified in terms of the mass of the liquids rather than the volumes, so that $y = \rho x$, where $\rho$ is the density of wine relative to that of water. The prior density $f_Y(y)\propto\frac{1}{(1+y)^2}$ would seem as appropriate as $f_X(x)\propto\frac{1}{(1+x)^2}$, yet it implies a different prior on $x$:

$$f_X(x) = f_Y(\rho x)\cdot\left|\frac{\operatorname{d}y}{\operatorname{d}x}\right| \propto \frac{\rho}{(1+\rho x)^2}$$

The same goes for no. molecules, or any extensive property of the liquids (& you needn't even know which is given in the statement of the problem). Require that

$$\Pr(c_1 < X < x) = \Pr(\rho c_1 < \rho X < \rho x)$$

so that the prior probability density must be a scale invariant function:

$$f_{\rho X}(\rho x) \propto f_X(x)$$

implying the power-law form

$$ f_X(x) = mx^r $$

Considering in addition the symmetry constraint, you can write

$$ m\left(\frac{1}{x}\right)^r = x^2 mx^r$$

& solve for the exponent: $r=-1$

The (normalized) density is therefore

$$ f_X = \frac{1}{x\cdot\log\left(\frac{c_2}{c_1}\right)}$$

(equivalent to a uniform prior on $\log(x)$), &

$$ \Pr(X < x) = \frac{\log\left(\frac{x}{c_1}\right)}{\log\left(\frac{c_2}{c_1}\right)}$$

Clearly the wine/water paradox has little relevance for a 'subjective' Bayesian approach; whatever prior distribution you elicit for $x$ should not of course conflict with one you elicit using an alternative parametrization, but there's no more to it than requiring coherent beliefs. The above is (something like) a Jaynesian^† solution, & 'objective' in the sense that different people will derive the same prior from the same information, regardless of personal beliefs—at least insofar as they agree on the constraints to apply. It's not claimed that all problems are soluble in this way (those that aren't you can dismiss as 'ill-posed'); & it seems to me that reasonable-seeming constraints might conflict, or that reasonable people might disagree about the need for a particular constraint.

Other 'objective' approaches involve choosing a prior distribution to maximize the influence of the data on the posterior distribution in some sense (e.g. the Jeffreys prior^‡). Naturally the choice requires a model of the data-generating process: priors thus chosen do not purport to have any epistemic significance in themselves—to represent the beliefs of someone in 'a state of ignorance' about the value of the parameter in question. (And they may or may not be in accord with constraints you think necessary on the form of the prior, which is quite all right.)

---

† Jaynes (2003), Probability Theory: The Logic of Science

‡ See the example in my answer to Do you have to adhere to the likelihood principle to be a Bayesian? & note that the Jeffreys prior for the probability parameter of a Bernoulli distribution changes according to the sampling scheme (binomial vs negative binomial).