This question was also asked in the link below but I don't think the answer there really addressed the question.
Latent output of variational autoencoder
So why the log variance and mean, why not log standard deviation and mean or standard deviation and mean?
Why not log standard deviation and mean?
It's simply because the normal distribution is frequently parameterized in terms of mean and variance. Beyond that, the choice was arbitrary. The log–standard deviation is just a rescaling of the log-variance; it's easy to go between the two: $\log \sigma^2 = 2 \log \sigma$.
Why not standard deviation and mean?
Emir's answer addresses this, and I'll explain from another angle. The TL;DR is that we need to make sure our model spits out a positive number for the variance, and modeling the log-variance was the easiest way with the tools in our toolbox.
Gradient descent is terrible at constrained optimization, e.g. trying to optimize $f(x, \theta)$ such that $\theta$ must be positive. Lagrange multipliers can introduce a soft constraint, but hard constraints are...well, hard. One way to get around this is to model the quantity we care about (e.g. $\theta$) as an invertible transformation of another quantity $g(\theta')$. There are no constraints on $\theta'$, but the range of $g$ is limited so that $\theta$ satisfies our constraint.
Here are some examples.
| Range | $g$ |
|---|---|
| $(0, 1)$ | $\text{sigmoid}(\cdot)$ |
| $(0, \infty)$ | $\exp(\cdot)$ |
| $(-\infty, \infty)$ | (no transformation needed) |
(The trick may be familiar to you if you've ever needed to go back and forth between the normal and natural parameterizations of exponential family distributions.)
We want to model a positive variance $\sigma^2 > 0$, which we enforce through $g = \exp$ instead of a constraint provided to the optimizer. Whatever value the log-variance $(\sigma^2)' = \log \sigma^2$ takes on, the variance will be a legal value (i.e. $> 0$).
If we learn the log variance or log standard deviation , then the variance would always be positive. This has to be guaranteed with network output. People usually output mean and variance together as a vector. However, mean is unconstrained compared to variance. That's why people use the log of variance/std.
You can also learn some parameter $\gamma$ with neural network alongside the mean $\mu$ and then transform the $\gamma$ to $\sigma$ via a softplus function(or smt else like simply applying log), a differentiable approximation to the ReLU function given as:
$$ f(x) = \log(1 + exp(x)) $$
All of them are same eventually. Then, to generate the latent variable, just invert the transformation. log is both for training stability and need for log sigma in ELBO.
Hope this helps
