How is the standard deviation of VAE's constructed?

Question

I am trying to build a Variational Autoencoder. I was looking at various codes online and found most of them in some way or another copy Francois Chollet (Google researchers) code.

Now my main question with this code is this part:

First, here's our encoder network, mapping inputs to our latent distribution parameters:

x = Input(batch_shape=(batch_size, original_dim))
h = Dense(intermediate_dim, activation='relu')(x)
z_mean = Dense(latent_dim)(h)
z_log_sigma = Dense(latent_dim)(h)

We can use these parameters to sample new similar points from the latent space:

def sampling(args):
    z_mean, z_log_sigma = args
    epsilon = K.random_normal(shape=(batch_size, latent_dim),
                              mean=0., std=epsilon_std)
    return z_mean + K.exp(z_log_sigma) * epsilon

# note that "output_shape" isn't necessary with the TensorFlow backend
# so you could write `Lambda(sampling)([z_mean, z_log_sigma])`
z = Lambda(sampling, output_shape=(latent_dim,))([z_mean, z_log_sigma])

As you can clearly see the $log(\sigma) = \mu$. Where did this assumption come from? How is it possible that we generate a random normal distribution with mean $\mu$ and standard deviation $\sigma$ like this?

Answer 1

The mean and log-variance are both encoded in the same way, but they have their own connections leading up to them from the encoding part $h$. Further on in the example, you will see that both z_mean and z_log_sigma are concatenated into a single layer.

I'm not too familiar with python, but in the same example given for the R interface to Keras, you can see this quite literally, as the function is called layer_concatenate.

z <- layer_concatenate(list(z_mean, z_log_var)) %>%
  layer_lambda(sampling)

(layer_lambda takes this concatenated input and evaluates the function sampling.)

Perhaps this explicit concatenation is not necessary in the python version due to differences in how R and python work.