How should one understand Bayes theorem with probability distributions?

Question

I'm learning about VAEs, and need to go this deep to understand them. However, the question is for Bayes theorem with probability distributions. I learnt about Bayes theorem from this video. Excellent explanation, with a simple example. There, are 4 events, a person is librarian ($P(H)$), or not ($P(\lnot H)$), and a person matches the description ($P(E)$), or not ($P(\lnot E)$). We want to know the probability of a person is librarian, given it matches the description. And we're calculating it in a nice visual way. It helps a lot that the probabilities are single numbers.

It is a lot harder to understand this with probability distributions, and I'm having a hard time trying to make up a real world example.

Or is it right to think about $P(H)$, and $P(\lnot H)$ together as a discrete probability distribution where on the x axis 0 is $P(\lnot H)$, and 1 is $P(H)$, and same with $E$?

In VAEs $p(z)$, and $q(z|x)$ are both normal probability distributions, and I can't really imagine a real world example similar.

Say example $p(z)$ is the probability of outcomes if two fair dices are rolled as in the picture:

What could $p(x|z)$ could be as a probability distribution? Say $z$ (the value of outcome) is 6, and we're checking the probability of one of the cubes is $x$ (6 possibilities).

However, in the VAE example, we know $x$ (in fact, that's the only thing we know for sure, and $p(z)$. Say, following for the above example, what's the probability of one of the cubes has the value of 6 ($x$), given ($z$).

Are those correct examples? If not, what could be an understandable, real world example, representing a VAE kind of situation with probabilities?

Answer 1

It is not necessary for $p(z)$ and $q(z|x)$ to be normal distributions in a VAE. We can have discrete VAEs too wherein $z$ are discrete. The distribution you've shown for $z$ is completely valid. It's a multinomial/categorical distribution. However, a mistake in your example is that you take $z$ to be observed (you know the value of the sum $S$ = $z$). In VAEs, $z$ is supposed to be latent/unobserved, and $x$ is observed. So I don't think your example is valid.

Here's my attempt at a real-world example using Discrete VAE. Let's say your data $X$ is the output of rolling a die, so $X = \{1, 3, 5, 6, 1, 4, 2, ...\}$. You want to figure out what the probabilities of the six faces are. The die is loaded, so it isn't just 1/6 for each face.

Now comes the modeling part. You assume that there is some hidden (latent) variable $z$ that determines how the die is loaded. If $z = 0$, then the die favors {1,2,3}. If $z = 1$, the die favors {4,5,6}. So the generative story is as follows

1. Sample $z$ from $p(z)$ = Bernoulli($\theta = 0.5$) - like a coin toss
2. Sample $x$ from $p(x|z)$ = Multinoulli($[p_{01}, p_{02}, p_{03}, p_{04}, p_{05}, p_{06}]$) if $z$ is 0 or Multinoulli($[p_11..p_16]$) if $z$ is 1 (just a vague model)

You'd expect that $[p_{01}, p_{02}, p_{03}]$ are higher as compared to $[p_{04}, p_{05}, p_{06}]$, and vice-versa for $z = 1$ case.

The parameters you learn with VAE-style training are 12 $p_{ij}$s for the decoder part. For encoder, you could approximate the posterior with $q_{\phi}(z | x) = Bern(\phi)$.

Effectively, you are gaining two things by modeling the die problem as above.

1. Given this data $X$, can I tell if my die is {1,2,3}-ish loaded or {4,5,6}-ish loaded? This comes from "Encoder" q(z | x). I'm being vague about what $z$ means because that's what it is. It's unknown. We just hope that it means something (like {4,5,6}-ish).
2. If I sample $z$ (0 or 1) - can I tell what $x$ I'd get? This is "decoding" -- $p(x|z)$. This gives me the power to "generate" data that looks almost similar to rolls of the die.

Another example - data $X$ = heights of students (continuous), and hidden $z$ could be the grades they belong to (discrete).

Check this answer for an understanding of continuous VAEs with a better example. Here, $x$ is an image (discrete pixel data of MNIST digits), and $z$ is continuous. The directions/vectors in z-space could be "italic", "bold" etc. Generative story for a variational auto encoder