Practical Bayes Theorem - how to use it?

Question

I am testing a variant of LDA (Latent Dirichlet Allocation) algorithm on some text data. Given some documents $d$ where each document is expressed as a distribution over words $w$, the algorithm effectively expresses the same documents over topics $z$. I am dropping the hyperparameters like the number of topics and the priors.

My dataset has 185311 documents with 26464 words in total. I picked the number of topics 10.

The algorithm computes the following:

• $p(w|z)$ - matrix of size (10 x 26464)
• $p(z)$ - matrix of size (1 x 10)
• $p(z|d)$ - matrix of size (185311 x 10)

For my experiments I need to compute $p(z|w)$. My very first idea was simply to use the columns of the $p(w|z)$ by normalizing to sum to one. After little thought I grew suspicious and recalled the Bayes theorem:

$p(z|w) = \frac{p(w|z)p(z)} {\sum_{i=1}^{K}p(w|z_i)p(z_i)} = \frac{p(w|z)p(z)} {p(w)}$

I understand that the $p(z|w)$ must be of size (26464 x 10) or (10 x 26464). Looking at the equation, I need $p(w|z)$ and $p(z)$ which I have. It looks like $p(z)$ serves as a kind of weighting while the whole thing in denominator makes sure that $p(z|w)$ is a probability distribution.

What I would do is to multiply elementwise the matrix $p(w|z)$ with a transposed matrix $p(z)$ and replicated along the columns to match the size of the first matrix. Then finally normalize the resulting matrix such that the sum along rows equals 1. Is this correct? Am I making a mistake somewhere?

I could easily write the expression but it is not clear for me how to practically compute it using the data I have. I am really willing to understand it; any pointers to some tutorials are welcome.

Answer 1

Here are the two probability matrices that you have to multiply together: $$ \begin{array}{cc} \boxed{\quad \quad \; \; p(z) \quad \;\; \tiny{1 \times 10}} & \boxed{ \begin{align} \\ \\ \quad \quad \quad p(w \mid z) \quad \quad \quad \\ \\ \tiny{10 \times 26464} \end{align} } \end{array} $$

After doing the multiplication, you get the joint probability $p(w, z)$ as follows ($.*$ denotes element-wise or Hadamard matrix product): $$ \begin{array}{ccccc} \underbrace{ \boxed{ \begin{array}{cccc} \boxed{ \begin{align} \\ \\ \hspace{-2mm}p(z) \hspace{-2mm} \\ \\ \hspace{-2mm}\tiny{10 \times 1} \end{align}} & \boxed{ \begin{align} \\ \\ \hspace{-2mm}p(z) \hspace{-2mm} \\ \\ \hspace{-2mm}\tiny{10 \times 1} \end{align}} & ... & \boxed{ \begin{align} \\ \\ \hspace{-2mm}p(z) \hspace{-2mm} \\ \\ \hspace{-2mm}\tiny{10 \times 1} \end{align}} \end{array} }}_{26464} & .* & \boxed{ \begin{align} \\ \\ \quad \quad \quad p(w \mid z) \quad \quad \quad \\ \\ \tiny{10 \times 26464} \end{align} } & = & \boxed{ \begin{align} \\ \\ \quad \quad \quad p(w, z) \quad \quad \quad \\ \\ \tiny{10 \times 26464} \end{align} } \end{array} $$ Finally, to get the conditional distribution $p(z | w)$ you just need to normalize the columns in $p(w, z)$.