1
$\begingroup$

I am trying to build a Variational Autoencoder. I was looking at various codes online and found most of them in some way or another copy Francois Chollet (Google researchers) code.

Now my main question with this code is this part:

First, here's our encoder network, mapping inputs to our latent distribution parameters:

x = Input(batch_shape=(batch_size, original_dim))
h = Dense(intermediate_dim, activation='relu')(x)
z_mean = Dense(latent_dim)(h)
z_log_sigma = Dense(latent_dim)(h)

We can use these parameters to sample new similar points from the latent space:

def sampling(args):
    z_mean, z_log_sigma = args
    epsilon = K.random_normal(shape=(batch_size, latent_dim),
                              mean=0., std=epsilon_std)
    return z_mean + K.exp(z_log_sigma) * epsilon

# note that "output_shape" isn't necessary with the TensorFlow backend
# so you could write `Lambda(sampling)([z_mean, z_log_sigma])`
z = Lambda(sampling, output_shape=(latent_dim,))([z_mean, z_log_sigma])

As you can clearly see the $log(\sigma) = \mu$. Where did this assumption come from? How is it possible that we generate a random normal distribution with mean $\mu$ and standard deviation $\sigma$ like this?

$\endgroup$
1
$\begingroup$

The mean and log-variance are both encoded in the same way, but they have their own connections leading up to them from the encoding part $h$. Further on in the example, you will see that both z_mean and z_log_sigma are concatenated into a single layer.

I'm not too familiar with python, but in the same example given for the R interface to Keras, you can see this quite literally, as the function is called layer_concatenate.

z <- layer_concatenate(list(z_mean, z_log_var)) %>%
  layer_lambda(sampling)

(layer_lambda takes this concatenated input and evaluates the function sampling.)

Perhaps this explicit concatenation is not necessary in the python version due to differences in how R and python work.

$\endgroup$
  • $\begingroup$ Yeah..I read it somewhere and yes i actually was not familiar with keras.. But can you tell something why is it not variance but log(variance)? $\endgroup$ – DuttaA May 28 '19 at 9:59
  • 2
    $\begingroup$ Good question! I searched the paper and couldn't find and explanation by the authors, but I assume it is because $\exp(x) > 0 \,\forall x$, and $\exp(\log(x)) = x$, so it is a way to ensure the variance is always positive. It might also just be for numerical stability, or convenience elsewhere in their derivations. $\endgroup$ – Frans Rodenburg May 28 '19 at 10:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.