6
$\begingroup$

Lately, after reading about GANs, I started experimenting with the MNIST dataset, and the result we acceptable. Here are some details about the networks I used:

  • Discriminator: 784 inputs $\rightarrow$ 256 ReLUs $\rightarrow$ 1 Sigmoids
  • Generator: 100 $\sim \mathcal{U}(-1,1)$ $\rightarrow$ 256 ReLUs $\rightarrow$ 784 Sigmoids

Using 20k examples of the MNIST dataset. Doing mini-batch (size = 100) gradient descent (using Tensorflow's AdamOptimizer). I trained the discriminator equally as the generator.

So I wanted to experiment more. I download a tiny dataset of 600 butterflies pictures, downscaled them to 64x64 and put them into grayscale, they look like this:

Dataset Sample

Given such tiny dataset, I expected the GAN to overfit the data and to generate images that are excessively close to what's in the dataset. I went for a topology similar to the one I used with MNIST:

  • Discriminator: 4096 inputs $\rightarrow$ 1024 ReLUs $\rightarrow$ 1 Sigmoids
  • Generator: 100 $\sim \mathcal{U}(-1,1)$ $\rightarrow$ 1024 ReLUs $\rightarrow$ 4096 Sigmoids

I also tried with hidden layers of size 512.

With minib-batches of 10 example, I tried training the discriminator: equally, twice as much, and three times as much as the generator.

Even after hours of training, the network still fails at generating butterfly images. Here are the usual results:

enter image description here enter image description here

What is possibly going wrong:

  • Is it the topology?
  • Is it the dataset size? Why didn't it overfit to the dataset?
  • Tried different training synchronization ratios, are there any other things to consider about training synchronization?
  • Activation functions?
$\endgroup$

3 Answers 3

3
$\begingroup$

Topology is not good enough. You may try with more hidden fully-connected layers, but what you really should do is to use a DCGAN, i.e., a GAN that uses convolutional layers. Try with at least 3 or 4 convolutional layers in both D and G. You need to know how CNNs work, though, but that's the way to go.

MNIST is composed of images that are a lot more simple than images in your data set. You don't even have to convert them to gray-scale, but you may, of course.

You should use batch normalization, but not on the first conv layer in D, and not on logits in G. Dropout is not necessary, but you may experiment with it.

Also, use Leaky ReLU instead of plain ReLU in GANs. You can select 0.2 as slope in the negative region.

Training 1:1 should be fine.

Batch size can be larger, like 32, 64, 128, but 10 is okay, too. That's not the problem.

For Adam optimizer, try with learning rate of 0.0002, and beta1 of 0.5.

When calculating D loss for real images, instead of 1.0 for labels, feed it with 0.9.

You can always train for more epochs, if it helps.

Data set size may be a problem, but try and see what you get.

Check out:

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

My project on GitHub that's similar to yours

My project contains what you need, but try to solve this problem on your own first. If you aren't familiar with CNNs, here's something to get you started:

http://cs231n.github.io/convolutional-networks/

https://en.wikipedia.org/wiki/Convolutional_neural_network

You can tell us how it went. ;)

$\endgroup$
2
$\begingroup$

I agree with Ivanbgd's answer, with the addition that you may then want to consider extensions to the basic GAN architecture, such as those which attempt to enforce a bijective mapping from the generated domain to the target domain e.g. CycleGAN, DiscoGAN, which are typically aimed at reducing the problem of mode collapse. Without this your GAN may learn to create one butterfly-esque image very well, and that is it.

It doesn't look like you are suffering from these issues yet but may be worth considering in the future. Below is a Github repo for a project I worked on to implement such GANs to achieve style transfer, which may be of interest to you. The basic GAN implemented in the project uses the CNN architecture outlined by Ivanbgd, in TensorFlow.

https://github.com/ZacKeskin/GAN-Factory

$\endgroup$
0
$\begingroup$

I know this answer comes a few years late, but it could be important. I have some recent findings on the original GAN theory, mainly Proposition 1 of the 2014 paper by Goodfellow et al GAN paper. I posted a detailed reponse on Quora and I can summarise here. The optimal discriminator result is only valid when the dimension of the data ($n_x$) is less than or equal to the dimension of the latent space ($n_z$). When $n_z < n_x$, the generator output PDF is singular (contains delta functions) and the variational argument used to obtain Proposition 1 breaks down. This is important because Proposition 1 is used in Proposition 2 - the GAN convergence result. A further result, which is a counter-example, shows that even in simple situations, the simplest being $n_z=n_x=1$ for a 1-dimensional GAN, one can get plateaux in the cost function where the stochastic gradient descent ascent algorithm stalls nowhere near the point where the generator PDF resembles the data PDF. Your simulations are using ($n_z=100$, $n_x=784$) for MNIST and ($n_z=100, n_x=4096$) for the butterly data set. Both have $n_z < n_x$ so the result I derived applies. Please refer to the paper for more details: Convergence & Optimality Analysis of low dimensional GANs.

To your specific questions on topology, dataset size, overfitting, training synchronization ratios, etc. There is currently no theory capable of answering such detailed questions on GAN training - only empirical evidence for certain data sets. Remember that you are optimising a very complicated non-linear cost function containing thousands of parameters. Because it is unsupervised, training a GAN is fundamentally harder than training a convnet. Convnet training also has direct supervision and uses equality constraints to reduce the number of degrees of freedom that produce feature maps. So Ivanbgd's suggestion to use convolutional layers in your network seems sensible.

GANs are not supervised and have no intrinsic parametric constraints, so there are a huge number of hyperparameter combinations, optimizers, batch sizes, ratios of generator / discriminator iterations to try and no guarantees of convergence. Many researches have proposed alternatives to the original GAN: they mainly focus on the form cost function. It is worth remembering that the cost function includes the expectation $E_x(D(x))$, which is the expectation of the discriminator output with respect to the data, whose PDF is unknown. You are replacing this expectation with sample-based Monte Carlo estimates but in a very high dimensional space. I would also question the use of just a uniform generator input distribution - you could try Gaussian.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.