How can Bayesians express true ignorance about a parameter and still perform inference? The binomial case

Question

How do I capture a claim of ignorance about a parameter in a Bayesian analysis?

For instance, suppose I observed a binomial random variable $X\sim Bin(n, p)$. Say $X = 5$ and $n = 10$. I want to make inference about $p$.

But now further suppose I don't know anything about $p$ beyond the fact that it is between zero and one. A uniform prior is not a claim of ignorance about $p$, as discussed here What is the Wine/Water Paradox in Bayesian statistics, and what is its resolution?.

So how can I express my ignorance about $p$ and proceed with inference given the observed data?

Answer 1

(Much of this answer is copied from another answer of mine here; I do so without further specific attribution of quoted or paraphrased material.)

A good way to proceed here is to conduct your analysis within the imprecise probability framework (see esp. Walley 1991, Walley 2000). In this framework the prior belief is represented by a set of probability distributions, and this leads to a corresponding set of posterior distributions. Below I will show you how this works for your specific example of binomial data.

Before getting to the implementation of this method, the first thing to note here is that the most extreme form of ignorance possible would be to take an imprecise prior distribution composed of the set of all possible priors on the parameter range. This would lead to an imprecise posterior with is the set of all possible posteriors, so your inference would then be vacuous. That is, total ignorance going in leads to total ignorance coming out. Consequently, if we want a useful inference at all, we must frame our "ignorance" in some way that restricts the imprecise prior to a reasonable range of priors from which a small set of posteriors can be formed. This can be done by allowing the prior expectation of the parameter to vary over all possible values in its range, but restricting the prior variance either to a single value or a small range to get a useful posterior inference.

---

Application to the binomial model: Suppose we observe data $X_1,...,X_n | \theta \sim \text{IID Bern}(\theta)$ where $\theta$ is the unknown parameter of interest. Usually we would use a beta density as the prior (both the Jeffrey's prior and reference prior are of this form). We can specify this form of prior density in terms of the prior mean $\mu$ and another parameter $\kappa > 1$ as:

$$\begin{equation} \begin{aligned} \pi_0(\theta | \mu, \kappa) = \text{Beta}(\theta | \mu, \kappa) = \text{Beta} \Big( \theta \Big| \alpha = \mu (\kappa - 1), \beta = (1-\mu) (\kappa - 1) \Big). \end{aligned} \end{equation}$$

(This form gives prior moments $\mathbb{E}(\theta) = \mu$ and $\mathbb{V}(\theta) = \mu(1-\mu) / \kappa$.) Now, in an imprecise model we could set the prior to consist of the set of all these prior distributions over all possible expected values, but with the other parameter fixed to control the precision over the range of mean values. For example, we might use the set of priors:

$$\mathscr{P}_0 \equiv \Big\{ \text{Beta}(\mu, \kappa) \Big| 0 \leqslant \mu \leqslant 1 \Big\}. \quad \quad \quad \quad \quad$$

Suppose we observe $s = \sum_{i=1}^n x_i$ positive indicators in the data. Then, using the updating rule for the Bernoulli-beta model, the corresponding posterior set is:

$$\mathscr{P}_\mathbf{x} = \Big\{ \text{Beta}\Big( \tfrac{s + \mu(\kappa-1)}{n + \kappa -1}, n+\kappa \Big) \Big| 0 \leqslant \mu \leqslant 1 \Big\}.$$

The range of possible values for the posterior expectation is:

$$\frac{s}{n + \kappa-1} \leqslant \mathbb{E}(\theta | \mathbb{x}) \leqslant \frac{s + \kappa-1}{n + \kappa-1}.$$

What is important here is that even though we started with a model that was "uninformative" with respect to the expected value of the parameter (the prior expectation ranged over all possible values), we nonetheless end up with posterior inferences that are informative with respect to the posterior expectation of the parameter (they now range over a narrower set of values). As $n \rightarrow \infty$ this range of values is squeezed down to a single point, which is the true value of $\theta$.

Answer 2

According to Bruno de Finetti, one of the major proponents of subjective Bayes, prior probabilities can be "elicited". This is a constructive act. De Finetti would hold that there is always some kind of belief about the future that can be expressed by prior probabilities. One way to do this is as follows: Before observing the data, imagine the following "game": You are forced to offer bets of x Kj (Kujambel, let's say that's the unit of money used here) on all kinds of possible outcomes for games in which you win 1 Kj in case that the outcome (or set of outcomes) occurs. The catch is that if you offer x Kj for winning 1 Kj for event $A$, your betting opponent can choose to either accept this, or to take 1-x for paying you 1 Kj in case that $A^c$ occurs. So you have an incentive to choose x not too high (because you may lose it if $A$ doesn't happen), but also not too low (because the opponent may then just take 1-x for $A^c$ rather than x for $A$).

Your prior can then be "reverse-engineered" from your bets. Actually this is not totally true - if you have $n=10$ and you assume exchangeable Bernoulli experiments (i.e., independent given $p$), there are not enough observable events to fully reconstruct the prior and you'd need in principle think about the case $n\to\infty$. However you could play this as a thought experiment imagining that at some point the true value of $p$ is revealed to you (as limit of outcomes of infinitely many Bernoulli experiments, say); the prior over $p$ translates into offered bets for all kinds of subsets of $[0,1]$ then. A probably sensible way to go on about this is to look at a number of supposedly "informationless" priors suggested in the literature, work out the resulting probabilities for all kinds of events of interest for $n=10$, and decide (all before having seen the data) which set of bets looks right for you.

"What if I'm not willing to bet anything?" - Well, it's a thought experiment, but if you refuse it anyway, de Finetti will not help you. Tough luck.

"I think this betting thing can get us somewhere, maybe you can prove that I always must have some probabilistic prior belief revealed by my behavior? I find that hard to prove without invoking strong assumptions though." This cannot be proved, rather one can see it as an implicit assumption or axiom. Personally I think of this as constructive, meaning that this is a scheme that will produce a prior that you can take as "yours" if you commit yourself to going with it, without the need to assume anything about beliefs that you have without being aware of them. Your choice whether you accept this or not.

Disclaimer: I am explaining de Finetti's approach here. I'm not claiming that this is the only correct one. This approach is controversial for various reasons, and it's not my aim to defend it against any objection I can imagine. However it is a clear principled approach that gets you somewhere.

Answer 3

Let's say we have parameters $\theta$ and some latent variable $z$. The product of all conditional dependencies is how we calculate the impact of all parameters.

$q\left(z \mid \theta_{\mathcal{O}}\right) \propto p(z) \prod_{k \in \mathcal{O}} q\left(z \mid \theta_{k}\right)$

If we would like to ignore the parameter $i$, then we just set $q(z \mid \theta_i)=1$