I am training a GAN using the following loss functions:

_, d_real_logit = discriminator(x_d)
_, d_fake_logit = discriminator(generator(z_g))
loss_d_real = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(labels=tf.ones_like(d_real_logit), logits=d_real_logit))
loss_d_fake = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(labels=tf.zeros_like(d_fake_logit), logits=d_fake_logit))
loss_d = loss_d_fake + loss_d_real

_, g_logit = discriminator(generator(z_g))
loss_g = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(labels = tf.ones_like(g_logit), logits=g_logit))

where discriminator is defined as:

def discriminator(x):
    y1_d = tf.nn.leaky_relu(tf.matmul(x, w_d_1) + b_d_1)
    y2_d = tf.nn.leaky_relu(tf.matmul(y1_d, w_d_2) + b_d_2)
    y3_d = tf.nn.leaky_relu(tf.matmul(y2_d, w_d_3) + b_d_3)
    y4_d = tf.nn.leaky_relu(tf.matmul(y3_d, w_d_4) + b_d_4)
    logits = tf.matmul(y4_d, w_d_5) + b_d_5
    y5_d = tf.nn.sigmoid(logits)

    return y5_d, logits

The plots of loss functions obtained are as follows: enter image description here

I understand that g_loss = 0.69 and d_loss = 1.38 are ideal situations, since that corresponds to discriminator output being 0.5 for both real and fake samples. But, for some reason the 2 loss values move away from these desired values as the training goes on. Does anyone know why this happens? (x axis is number of epochs/100)

  • $\begingroup$ I'm also having trouble understanding g_loss/d_loss in my GAN. $\endgroup$ Mar 12 '19 at 21:58

The Discriminator and generator in a GAN training scheme work one against the other, so naturally when one improves, the other should deteriorate (It is not a perfect -1 correlation but the 2 losses are correlated).

The task of the Generator is to create a fake signal (usually image) which is indistinguishable from a real signal.

The task of the Discriminator is to distinguish between 2 cases - real and fake signals.

The task of binary classification is considerably simpler (and probably has a smaller feature space) than generating a signal. In addition in the initial epochs, the generated signal is far from a real one so the discriminator gets good loss values.

Since the task of the generator is difficult (and far from being convex), it is initially hard for it to find a good gradient to follow during training, so during the first epochs of the training, the generator loss displays somewhat random behavior.

Eventually the generator begins to improve (in the given image it happens after approximately 50 epochs), so the task of the discriminator becomes harder and its performance deteriorates. This is a good sign that the training scheme is working.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.