I'm learning about VAEs, and need to go this deep to understand them. However, the question is for Bayes theorem with probability distributions. I learnt about Bayes theorem from this video. Excellent explanation, with a simple example. There, are 4 events, a person is librarian ($P(H)$), or not ($P(\lnot H)$), and a person matches the description ($P(E)$), or not ($P(\lnot E)$). We want to know the probability of a person is librarian, given it matches the description. And we're calculating it in a nice visual way. It helps a lot that the probabilities are single numbers.

It is a lot harder to understand this with probability distributions, and I'm having a hard time trying to make up a real world example.

Or is it right to think about $P(H)$, and $P(\lnot H)$ together as a discrete probability distribution where on the x axis 0 is $P(\lnot H)$, and 1 is $P(H)$, and same with $E$?

In VAEs $p(z)$, and $q(z|x)$ are both normal probability distributions, and I can't really imagine a real world example similar.

Say example $p(z)$ is the probability of outcomes if two fair dices are rolled as in the picture:

enter image description here

What could $p(x|z)$ could be as a probability distribution? Say $z$ (the value of outcome) is 6, and we're checking the probability of one of the cubes is $x$ (6 possibilities).

However, in the VAE example, we know $x$ (in fact, that's the only thing we know for sure, and $p(z)$. Say, following for the above example, what's the probability of one of the cubes has the value of 6 ($x$), given ($z$).

Are those correct examples? If not, what could be an understandable, real world example, representing a VAE kind of situation with probabilities?


1 Answer 1


It is not necessary for $p(z)$ and $q(z|x)$ to be normal distributions in a VAE. We can have discrete VAEs too wherein $z$ are discrete. The distribution you've shown for $z$ is completely valid. It's a multinomial/categorical distribution. However, a mistake in your example is that you take $z$ to be observed (you know the value of the sum $S$ = $z$). In VAEs, $z$ is supposed to be latent/unobserved, and $x$ is observed. So I don't think your example is valid.

Here's my attempt at a real-world example using Discrete VAE. Let's say your data $X$ is the output of rolling a die, so $X = \{1, 3, 5, 6, 1, 4, 2, ...\}$. You want to figure out what the probabilities of the six faces are. The die is loaded, so it isn't just 1/6 for each face.

Now comes the modeling part. You assume that there is some hidden (latent) variable $z$ that determines how the die is loaded. If $z = 0$, then the die favors {1,2,3}. If $z = 1$, the die favors {4,5,6}. So the generative story is as follows

  1. Sample $z$ from $p(z)$ = Bernoulli($\theta = 0.5$) - like a coin toss
  2. Sample $x$ from $p(x|z)$ = Multinoulli($[p_{01}, p_{02}, p_{03}, p_{04}, p_{05}, p_{06}]$) if $z$ is 0 or Multinoulli($[p_11..p_16]$) if $z$ is 1 (just a vague model)

You'd expect that $[p_{01}, p_{02}, p_{03}]$ are higher as compared to $[p_{04}, p_{05}, p_{06}]$, and vice-versa for $z = 1$ case.

The parameters you learn with VAE-style training are 12 $p_{ij}$s for the decoder part. For encoder, you could approximate the posterior with $q_{\phi}(z | x) = Bern(\phi)$.

Effectively, you are gaining two things by modeling the die problem as above.

  1. Given this data $X$, can I tell if my die is {1,2,3}-ish loaded or {4,5,6}-ish loaded? This comes from "Encoder" q(z | x). I'm being vague about what $z$ means because that's what it is. It's unknown. We just hope that it means something (like {4,5,6}-ish).
  2. If I sample $z$ (0 or 1) - can I tell what $x$ I'd get? This is "decoding" -- $p(x|z)$. This gives me the power to "generate" data that looks almost similar to rolls of the die.

Another example - data $X$ = heights of students (continuous), and hidden $z$ could be the grades they belong to (discrete).

Check this answer for an understanding of continuous VAEs with a better example. Here, $x$ is an image (discrete pixel data of MNIST digits), and $z$ is continuous. The directions/vectors in z-space could be "italic", "bold" etc. Generative story for a variational auto encoder

  • $\begingroup$ Excellent explanation, much appreciated. According to your explanation I just got confused about what does $X$ mean in a training step. When we input a single $X$ from a batch to training, do we look at it as $X_i$ from the possible $X$s, or it is $X$ itself containing the possible $X_i$s inside? To follow your answer, do we input ${1,3,5,6...}$ as $X$ from a batch (so there are multiple like this in the batch), or it is the batch itself, and we input 1,3,5,6 separately? $\endgroup$ Commented Dec 27, 2020 at 13:26
  • 1
    $\begingroup$ Thank you :) $X$ is the entire dataset; $x_i$ would be a single training example ($x_i$ = 3, $x_i$ = 1), etc. You could batch it too. Note that categorical variables are often one-hot encoded - so when you batch it, your input will be of shape $(B, 6)$, as in, $3 = [0, 0, 1, 0, 0, 0]$. As I mentioned, this is a vague model and probably not of much use - since the decoder is strong enough to model $p(X)$. There is something called posterior collapse in VAEs, wherein the decoder is so strong that it doesn't even use $z$. That is bad - because we want to know the unseen hidden variable $z$ info $\endgroup$
    – stbv
    Commented Dec 27, 2020 at 17:54
  • $\begingroup$ In the case of MNIST images a single image is $x_i$, and the whole dataset (all batches) is $X$ right? Otherwise it doesn't make too much sense to me, and will be the case when I think I understand the process, then realise that I don't $\endgroup$ Commented Dec 27, 2020 at 19:14
  • $\begingroup$ Yes, a single image is $x_i$. The image is represented as a matrix though - of size (32, 32, 3) for example - the RGB pixels of a 32*32 image. It is later flattened into a 32*32*3 vector. Each pixel can take values from [0...255] assuming 8-bits. So it's as if each image is equivalent to 32*32*3 rolls of a 256-faced die 😅 enormous complexity! So don't confuse a die with an image. $\endgroup$
    – stbv
    Commented Dec 27, 2020 at 21:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.