Bayesian inference out of partial information - Dirichlet example

Suppose we have two coins $$X_1$$ and $$X_2$$. They are possibly biased and correlated coins. The heads probability of each coins is denoted by $$p_1$$ and $$p_2$$ which we don't know at the beginning. The objective is to estimate the probabilities of HH, TT, H|H, H|T or so on as well as estimating $$p_1$$ and $$p_2$$.

To begin, I assume that $$p=(p_{HH},p_{HT},p_{TH},p_{TT})\sim Dir(a_{HH},a_{HT},a_{TH},a_{TT}).$$

If experiments (tossings of two coins, in this case) run in a perfect condition, it would be easy to make a Bayesian inference out of the experiments: we just update the parameter $$a$$.

However, suppose that only one coins, say coin 1, is tossed and the outcome is "H". What would the posterior look like in this case?

I think one of my three attempts should be correct, but I'm not sure which one should be.

Answer 1: $$X_2$$ is "H" with probability $$p_2$$. And the expected value of $$p_2$$ is $$\frac{a_{HH}+a_{TH}}{a_{HH}+a_{HT}+a_{TH}+a_{TT}}$$. Let $$q$$ denote this expected value. The posterior is $$(p_{HH},p_{HT},p_{TH},p_{TT})\sim qDir(a_{HH}+1,a_{HT},a_{TH},a_{TT})+(1-q)Dir(a_{HH},a_{HT}+1,a_{TH},a_{TT})$$ beacause $$(HH)$$ happens with probability $$q$$ and $$(HT)$$ occurs with the other probability.

Answer 2: Maybe I can't use the expected value to marginalize the missing information. So, let $$p^1\sim Dir(a_{HH}+1,a_{HT},a_{TH},a_{TT})$$ and $$p^2\sim Dir(a_{HH},a_{HT}+1,a_{TH},a_{TT})$$. Then, $$p =_D p_{HH}p^1+p_{HT}p^2,$$ where $$=_D$$ denotes the equality in distribution.

Answer 3: $$X_1$$ is head with probability $$p_{HH}+p_{HT}$$, so by the Bayes rule,

$$\begin{eqnarray*} f(p|X_1=H)&\propto& f(X_1=H|p)f(p)\\&=&(p_{HH}+p_{HT})f(p). \end{eqnarray*}$$

Obviously, the two answers lead me to different posteriors.. Could anyone tell me which one should be correct?

• yeah, that really answers my question. Thanks – Andeanlll Oct 3 '19 at 2:21

Suppose both coins are tossed once but we only observe one of them. Assume the coin we get to observe is independent of the outcome of the toss.

Let $$X_1$$ and $$X_2$$ be random variables representing the outcome for each coin. And, let $$\theta = [p_{HH}, p_{HT}, p_{TH}, p_{TT}]$$ be a parameter vector indicating the joint probability of each possible pair of outcomes:

$$p(x_1, x_2 \mid \theta) = \left\{ \begin{array}{cl} p_{HH} & x_1=H, x_2=H \\ p_{HT} & x_1=H, x_2=T \\ p_{TH} & x_1=T, x_2=H \\ p_{TT} & x_1=T, x_2=T \\ \end{array} \right.$$

Suppose we've only observed the first coin. Then, the goal is to compute the posterior distribution over parameters, given the observed outcome:

$$p(\theta \mid x_1) \propto p(x_1 \mid \theta) p(\theta)$$

where $$p(\theta)$$ is the prior (which you've indicated is Dirichlet). $$p(x_1 \mid \theta)$$ is the observed data likelihood, which is obtained by marginalizing the missing outcome out of the complete data likeilhood $$p(x_1, x_2 \mid \theta)$$:

$$p(x_1 \mid \theta) = \sum_{x_2} p(x_1, x_2 \mid \theta)$$

So, if $$X_1=H$$, then $$p(x_1 \mid \theta) = p_{HH} + p_{HT}$$, and the posterior would be:

$$p(\theta \mid x_1) \propto (p_{HH} + p_{HT}) p(\theta)$$

Alternatively, if $$X_1=T$$, then $$p(x_1 \mid \theta) = p_{TH} + p_{TT}$$, and the posterior would be:

$$p(\theta \mid x_1) \propto (p_{TH} + p_{TT}) p(\theta)$$

If I understand your notation correctly, this is consistent with your 'answer 3'. A similar procedure can be used to compute $$p(\theta \mid x_2)$$ if only the second coin is observed.