To help me understand some concepts I'm learning in my first exposition to machine learning, I'm trying to tackle the following "simple" problem
The setup of the problem is as follows:
- My friend has a coin and two 6-sided dice (all possibly biased)
- He first tosses a coin, if the result is heads, he throws dice A, otherwise he throws dice B
- He repeats this $n$ times and gives me the results of each die throw (without telling me which die generated which result)
How can I estimate the bias of the coin and the bias of each die using only the data he gives me?
My work so far:
- Bias of the coin: $\theta$
- Bias of die $k$: $\theta^k$ (6-dimensional vector)
- Bias of side $i$ of die $k$: $\theta_i^k$ (e.g. the probability of landing a 3 with die A is $\theta_3^A$)
- $i$'th unobserved coin toss result (with value either A or B): $z_i$
- All the coin tosses results: $\mathbf{z}$
- $i$'th dice throw result: $w_i$
- All the dice throws results: $\mathbf{w}$
- I want to compute $P(\theta,\theta^A,\theta^B~|~\mathbf{w})$. To do that, I use bayes' rule:
$$ P(\theta,\theta^A,\theta^B~|~\mathbf{w}) = \frac{P(\mathbf{w}~|~\theta,\theta^A,\theta^B)\cdot P(\theta,\theta^A,\theta^B)}{P(\mathbf{w})}$$
- To compute $P(\mathbf{w}~|~\theta,\theta^A,\theta^B)$ I first note that the throws are independent of each other, so
$$P(\mathbf{w}~|~\theta,\theta^A,\theta^B)=\prod_{i=1}^nP(w_i~|~\theta,\theta^A,\theta^B)$$ and then I apply the law of total probability
$$ P(w_i~|~\theta,\theta^A,\theta^B) = P(w_i~|~z_i=A,\theta,\theta^A,\theta^B)\cdot P(z_i=A~|~\theta,\theta^A,\theta^B) + P(w_i~|~z_i=B,\theta,\theta^A,\theta^B)\cdot P(z_i=B~|~\theta,\theta^A,\theta^B)$$
These terms can now be directly computed as
$$P(w_i~|~\theta,\theta^A,\theta^B)= \left( \prod_{j=1}^6 \theta_j^{A[w_i=j]}\right)\theta + \left( \prod_{j=1}^6 \theta_j^{B[w_i=j]}\right)(1-\theta)$$ (where $[x=i]$ is the Iverson Bracket)
Now, in order to compute $P(\theta,\theta^A,\theta^B)$, I assume three independent priors, so $P(\theta,\theta^A,\theta^B) = P(\theta)\cdot P(\theta^A)\cdot P(\theta^B) $. For the coin I'll use a uniform Beta distribution and for the dice a uniform Dirichlet distribution (one for each).
Finally, I could use the law of total probability again to compute $P(\mathbf{w})$, but I believe this to be intractable. Instead, I'm trying to understand how I can apply Gibbs sampling to this problem and estimate the bias parameters without computing $P(\mathbf{w})$.
If anything needs clarification, please say so. Any help would be appreciated, thanks in advance.
