If we have a conditional categorical distribution, with unknown parameters, we can represent with a table, as in the example below:

\begin{align*} &z \quad P(z|\theta)\\ &0 \quad \theta_0\\ &1 \quad \theta_1 \end{align*} where $\theta = [\theta_{0},\theta_{1}]$.

How would this representation generalise a categorical variable (also with unknown parameters) which is also dependent on another categorical variable? If we simply define it as follows:

\begin{align*} &x \quad z \quad P(z|\theta, x)\\ &0 \quad 0\quad \theta_0\\ &0 \quad 1\quad \theta_1\\ &1 \quad 0\quad \theta_2\\ &1 \quad 1\quad \theta_3\\ \end{align*}

with $\theta = [\theta_0, \theta_1, \theta_2, \theta_3]$

we cannot enforce that $\theta_0+\theta_1=1$ and $\theta_2+\theta_3=1$, which must be true for it to be a valid conditional distribution. An example of this type of structure can be found in the Author-Topic Model (extension to LDA) which is depicted below.

One solution could maybe be to define the conditional distribution as follows:

\begin{align*} &x \quad z \quad P(z|\theta_{x0}, \theta_{x1}, x)\\ &0 \quad 0\quad \theta_{00}\\ &0 \quad 1\quad \theta_{01}\\ &1 \quad 0\quad \theta_{10}\\ &1 \quad 1\quad \theta_{11}\\ \end{align*}

where $\theta_{x0} = [\theta_{00},\theta_{01}]$ and $\theta_{x1} = [\theta_{10},\theta_{11}]$. Then we can have two independent Dirichlet priors over the parameters of the conditional distribution. I am however not sure if this is the best solution or even if it would be correct?

enter image description here Image from: 'The Author-Topic Model for Authors and Documents' (Rosen-Zvi, Griffiths, Steyvers, Smyth)


Your notation isn't helping here, as you have a $\theta_{0}$ and $\theta_{1}$ in both expressions which aren't directly related.

I think it probably makes most sense to think about the joint distribution $p(x=i, z=j)$, which is given by $q_{ij}$, and thus parametrised through a 4-vector with all positive entries which sums to unity.

As your parameters are unknown, I guess you're trying to infer them, and you'll do Dirichlet inference and get $p(\underline{q})=D(\underline{q};\underline{\alpha})$, a dirichlet distribution parametrised by a vector $\alpha$

If you want instead to do inference on $p(x=i)$, that's the same as inference on the sum $p(x=i, z=0)+p(x=i, x=1)$. In terms of your dirichlet distribution, you have a joint distribution for four random variables, and you want to calculate the distribution of the sum of two of them, which requires integrating out the other two degrees of freedom.

I've heard the result of this calculation referred to as the "aggregation property" of the Dirichlet Distribution. For more details of the calculation, see https://stats.stackexchange.com/questions/372295/distribution-of-conditional-frequencies-when-frequencies-follow-a-dirichlet-di/ (and in particular my answer)

In short, if the probabilities of (x,z)=(0,0),(0,1),(1,0),(1,1) respectively is parametrised by $q_{1},q_{2},q_{3},q_{4}$, and by some inference process, your posterior on these parameters is a Dirichlet distribution parametrised through a vector $\underline{\alpha}$, then you can show that if you parametrise the probability of $z=0$ through $\theta$ (and thus by conservation, $z=1$ by $1-\theta$), then your posterior for $\theta$, will be given by $\frac{1}{B(\alpha_{1}+\alpha{3}, \alpha_{2}+\alpha_{4})}\theta ^{\alpha_{1}+\alpha_{3}-1}(1-\theta)^{\alpha_{2}+\alpha_{4}}$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.