15
$\begingroup$

It is difficult to ask this question succinctly in the title, so let me explain.

From all the examples of VAEs I have seen, there seem to be 2 approaches used to implement them.

  1. In these, approaches, the output of the decoder is a distribution. Specifically, a Bernoulli distribution. The loss is related to the likelihood of the input data with respect to this distribution. When generating new variables, not only are samples drawn from the latent space distributions, but also, samples are drawn from the output distributions. I have a question related to the parameterization of the output distributions using this approach, but I will save that for the end.

  2. In these, approaches, the output of the decoder is the actual predicted data points, which are then compared to the input to calculate the loss, as mean squared error. The only thing that ever gets sampled is the latent space.

These 2 approaches are fundamentally different. I notice that in the first approach, the authors relate the idea to a statistical/probabilistic motivation, and it seems to have a solid theoretical foundation. For this reason I am inclined to adopt the 1st approach.

So my questions are:

  1. Are either of these approaches considered to be superior?
  2. The second question is more of a sense check: Bernoulli distribution makes sense for grey scale pixels. But for something like stock returns, we would want some other distribution, such as a t-distribution. In this case, the output of the encoder would be 3 vectors, 1 for location, 1 for scale, and 1 for degrees of freedom, right?
$\endgroup$

1 Answer 1

12
$\begingroup$

Question 1:

The output of the decoder aims to model the distribution $p(x|t)$, i.e. the distribution of data $x$ given the latent variable $t$. Therefore, in principle, it should always be probabilistic.

However, in some cases, people simply use the mean squared error as the loss and, as you said, the output of the decoder is the actual predicted data points. Note that this approach can also be viewed as probabilistic, in the sense that it is is equivalent to modeling $p(x|t)$ as Gaussian with identity covariance, $p(x|t) = \mathcal{N}(x|\mu(t), I)$. In this case, the output of the decoder is the mean $\mu(t)$ and, therefore, for an example $x_i$, you get the following reconstruction loss:

\begin{align} -\log(p(x_i | t_i)) &= -\log \left(\frac{1}{\sqrt{(2\pi)^k |I|}} \exp \left(-\frac{1}{2}(x_i-\mu(t_i))^\intercal I (x_i-\mu(t_i))\right)\right) \\ &= \frac{1}{2}||x_i - \mu(t_i)||^2 + \text{const.} \end{align}

which, as you can see, is proportional to the mean squared error (plus some constant).

Question 2:

Bernoulli distribution makes sense for grey scale pixels.

This is not quite true. The correct statement would be that Bernoulli distribution makes sense for black and white (i.e. binary) images. The Bernoulli distribution is binary, so it assumes that observations may only have two possible outcomes. It is true that people sometimes use it for grayscale images, but this is an abusive interpretation of the VAE. It may work pretty well for datasets that are almost black and white, like MNIST. However, a binarized version of the MNIST dataset exists and, in rigor, this is the version that should be used together with a Bernoulli VAE.

But for something like stock returns, we would want some other distribution, such as a t-distribution. In this case, the output of the encoder would be 3 vectors, 1 for location, 1 for scale, and 1 for degrees of freedom, right?

I would try a Gaussian first, $p(x|t) = \mathcal{N}(x|\mu(t), \sigma^2(t))$, so the decoder would output two values, $\mu(t)$ and $\sigma^2(t)$. But yeah, if you really want a t-distribution, then that is the way to go.

$\endgroup$
5
  • $\begingroup$ Could you please explain why Bernoulli distribution makes sense only for black and white images? I understand what Bernoulli distribution does, but we are choosing a distribution for the latent space, not for the data samples space (which is binary), so I don’t find it quite evident why it makes sense to select this distribution for the latent space. Also, I have seen in some papers (e.g. $\beta$-VAE) that Bernoulli distribution is used for non-binary datasets. $\endgroup$
    – nim.py
    Dec 10, 2020 at 11:20
  • $\begingroup$ I meant that a Bernoulli posterior $p(x|t)$ makes sense for black and white images, i.e. those where $x_i \in \{0,1\}$. I was not talking about the posterior distribution on the latent space $p(t|x)$, which I was assuming to be Gaussian (as in "vanilla VAE"), but this need not be the case. $\endgroup$
    – learner
    Dec 11, 2020 at 13:06
  • $\begingroup$ In the second case, N(x|mu, sigma^2), the expression for -log(p(x|t)) would include the terms 1/sigma^2 * ||x - mu||^2 + log(sigma). So the loss function would change. But what I don't understand is, how to output sigma from the decoder? Is it similar to how encoder outputs mu_z, sigma_z ? or there is some other technique? $\endgroup$
    – user62198
    Feb 20, 2021 at 2:29
  • $\begingroup$ You could that, but, in that case, the optimization problem is ill-posed (see stats.stackexchange.com/questions/373858/…). Alternatively, you can simply set the variance to a fixed value, as a hyperparameter. $\endgroup$
    – learner
    Feb 22, 2021 at 13:03
  • $\begingroup$ This NeurIPS '19 paper is all about the Bernoulli black/white/grayscale thing and proposes a simple solution: using a ContinuousBernoulli output distribution, which has proper support in [0, 1] (and not just in {0,1} like Bernoulli). $\endgroup$
    – Eike P.
    Sep 19, 2023 at 22:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.