3
$\begingroup$

I recently worked on Variational(VAE) Autoencoder. For me, Autoencoder works like this:

enter image description here

In the loss function, we have

  • a reconstruction error term which allows us to minimize the distance between input and output.
  • a regularization term which allows us to match the latent variables(z1,z2) to a unit gaussian distribution.

But does someone know why do we introduce the random effect? To match a gaussian distribution, why not just use a gaussian likehood as a regularization term of loss function?

The answer might be from a bayes prospective of VAE autoencoder? Even a small hint might be of great help, thanks!

$\endgroup$

2 Answers 2

4
$\begingroup$

As shown in the figure, the encoder produces $\mu$ and $\sigma$, which are the mean and standard deviation of the posterior distribution $q(z|x)$.

The random effect comes from drawing samples from the posterior distribution. Each sample $z$ can be obtained by $z = \mu+\sigma\epsilon$, where $\epsilon$ is a sample from a Gaussian distribution with zero mean and unit variance, which can be easily obtained.

The reason why we need the randomness is, the reconstruction error term in the loss function is actually an expectation over the posterior distribution which is difficult to solve analytically, so we use sample mean to approximate it.

For the derivation of the regularization term you can refer to the original paper https://arxiv.org/abs/1312.6114

$\endgroup$
0
1
$\begingroup$

The data to which you fit the VAE has noise. If you fit a model that allows no noise (randomness) then this randomness will no where to go but into the coefficients (weights). A perfect model will allow for the noise to find its resting place in the randomness (noise) component of the model, and let the weights to set on the correct values.

$\endgroup$
2
  • $\begingroup$ is there a paper that run ablation study/proves your hypothesis? My understanding was that random component allows us to model unseen data points (those that are slightly different from observed ones). $\endgroup$ May 4, 2023 at 16:27
  • $\begingroup$ @PavelPodlipensky I don't think there's a specific paper that shows what I said. it's rather understood that there's noise in data, and it needs a component to capture it $\endgroup$
    – Aksakal
    May 8, 2023 at 1:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.