Were generative adversarial networks introduced by Jürgen Schmidhuber?

Question

I read on https://en.wikipedia.org/wiki/Generative_adversarial_networks :

[Generative adversarial networks] were introduced by Ian Goodfellow et al in 2014.

but Jurgen Schmidhuber claims to have performed similar work earlier in that direction (e.g., there was some debate at NIPS 2016 during the generative adversarial networks tutorial: https://channel9.msdn.com/Events/Neural-Information-Processing-Systems-Conference/Neural-Information-Processing-Systems-Conference-NIPS-2016/Generative-Adversarial-Networks see 1h03min).

Was the idea behind generative adversarial networks first publicly introduced by Jürgen Schmidhuber? If not, how similar were Jürgen Schmidhuber's ideas?

Answer 1

I self-published the basic idea of a deterministic variety of generative adversarial networks (GANs) in a 2010 blog post (archive.org). I had searched for but could not find anything similar anywhere, and had no time to try implementing it. I was not and still am not a neural network researcher and have no connections in the field. I'll copy-paste the blog post here:

2010-02-24

A method for training artificial neural networks to generate missing data within a variable context. As the idea is hard to put in a single sentence, I will use an example:

An image may have missing pixels (let's say, under a smudge). How can one restore the missing pixels, knowing only the surrounding pixels? One approach would be a "generator" neural network that, given the surrounding pixels as input, generates the missing pixels.

But how to train such a network? One can't expect the network to exactly produce the missing pixels. Imagine, for example, that the missing data is a patch of grass. One could teach the network with a bunch of images of lawns, with portions removed. The teacher knows the data that is missing, and could score the network according to the root mean square difference (RMSD) between the generated patch of grass and the original data. The problem is that if the generator encounters an image that is not part of the training set, it would be impossible for the neural network to put all the leaves, especially in the middle of the patch, in exactly the right places. The lowest RMSD error would probably be achieved by the network filling the middle area of the patch with a solid color that is the average of the color of pixels in typical images of grass. If the network tried to generate grass that looks convincing to a human and as such fulfills its purpose, there would be an unfortunate penalty by the RMSD metric.

My idea is this (see figure below): Train simultaneously with the generator a classifier network that is given, in random or alternating sequence, generated and original data. The classifier then has to guess, in the context of the surrounding image context, whether the input is original (1) or generated (0). The generator network is simultaneously trying to get a high score (1) from the classifier. The outcome, hopefully, is that both networks start out really simple, and progress towards generating and recognizing more and more advanced features, approaching and possibly defeating human's ability to discern between the generated data and the original. If multiple training samples are considered for each score, then RMSD is the correct error metric to use, as this will encourage the classifier network to output probabilities.

Artificial neural network training setup

When I mention RMSD at the end I mean the error metric for the "probability estimate", not the pixel values.

I originally started considering the use of neural networks in 2000 (comp.dsp post) to generate missing high frequencies for up-sampled (resampled to a higher sampling frequency) digital audio, in a way that would be convincing rather than accurate. In 2001 I collected an audio library for the training. Here are parts of an EFNet #musicdsp Internet Relay Chat (IRC) log from 20 January 2006 in which I (yehar) talk about the idea with another user (_Beta):

[22:18] <yehar> the problem with samples is that if you don't have something "up there" already then what can you do if you upsample...
[22:22] <yehar> i once collected a big library of sounds so that i could develop a "smart" algo to solve this exact problem
[22:22] <yehar> i would have used neural networks
[22:22] <yehar> but i didn't finish the job :-D
[22:23] <_Beta> problem with neural networks is that you have to have some way of measuring the goodness of results
[22:24] <yehar> beta: i have this idea that you can develop a "listener" at the same time as you develop the "smart up-there sound creator"
[22:26] <yehar> beta: and this listener will learn to detect when it's listening a created or a natural up-there spectrum. and the creator develops at the same time to try to circumvent this detection

Sometime between 2006 and 2010, a friend invited an expert to take a look at my idea and discuss it with me. They thought that it was interesting, but said that it wasn't cost-effective to train two networks when a single network can do the job. I was never sure if they did not get the core idea or if they immediately saw a way to formulate it as a single network, perhaps with a bottleneck somewhere in the topology to separate it into two parts. This was at a time when I didn't even know that backpropagation is still the de-facto training method (learned that making videos in the Deep Dream craze of 2015). Over the years I had talked about my idea with a couple of data scientists and others that I thought might be interested, but the response was mild.

In May 2017 I saw Ian Goodfellow's tutorial presentation on YouTube [Mirror], which totally made my day. It appeared to me as the same basic idea, with differences as I currently understand outlined below, and the hard work had been done to make it give good results. Also he gave a theory, or based everything on a theory, of why it should work, while I never did any sort of a formal analysis of my idea. Goodfellow's presentation answered questions that I had had and much more.

Goodfellow's GAN and his suggested extensions include a noise source in the generator. I never thought of including a noise source but have instead the training data context, better matching the idea to a conditional GAN (cGAN) without a noise vector input and with the model conditioned on a part of the data. My current understanding based on Mathieu et al. 2016 is that a noise source is not needed for useful results if there is enough input variability. The other difference is that Goodfellow's GAN minimizes log-likelihood. Later, a least squares GAN (LSGAN) has been introduced (Mao et al. 2017) which matches my RMSD suggestion. So, my idea would match that of a conditional least squares generative adversarial network (cLSGAN) without a noise vector input to the generator and with a part of the data as the conditioning input. A generative generator samples from an approximation of the data distribution. I do now know if and doubt that real-world noisy input would enable that with my idea, but that is not to say that the results would not be useful if it didn't.

The differences mentioned in the above are the primary reason why I believe Goodfellow did not know or hear about my idea. Another is that my blog has had no other machine learning content, so it would have enjoyed very limited exposure in machine learning circles.

It is a conflict of interests when a reviewer puts pressure on an author to cite the reviewer's own work.

Answer 2

An answer from Ian Goodfellow on Was Jürgen Schmidhuber right when he claimed credit for GANs at NIPS 2016? posted on 2017-03-21:

He isn’t claiming credit for GANs, exactly. It’s more complicated.

You can see what he wrote in his own words when he was a reviewer of the NIPS 2014 submission on GANs: Export Reviews, Discussions, Author Feedback and Meta-Reviews (mirror)

He’s the reviewer that asked us to change the name of GANs to “inverse PM.”

Here's the paper he believes is not being sufficiently acknowledged: http://ftp://ftp.idsia.ch/pub/juergen/factorial.pdf (mirror)

I don’t like that there is no good way to have issues like this adjudicated. I contacted the NIPS organizers and asked if there is a way for Jürgen to file a complaint about me and have a committee of NIPS representatives judge whether my publication treats his unfairly. They said there is no such process available.

I personally don’t think that there is any significant connection between predictability minimization and GANs. I have never had any problem acknowledging connections between GANs and other algorithms that actually are related, like noise-contrastive estimation and self-supervised boosting.

Jürgen and I intend to write a paper together soon describing the similarities and differences between PM and GANs, assuming we’re able to agree on what those are.

Answer 3

This is taken straight out from the 1991 original paper of Schmidhuber in 1991:

"I propose a novel general principle for unsupervised learning of distributed non-redundant internal representations of input patterns. The principle is based on two opposing forces. For each representational unit there is an adaptive predictor which tries to predict the unit from the remaining units. In turn, each unit tries to react to the environment such that it minimizes its predictability. This encourages each unit to filter `abstract concepts' out of the environmental input such that these concepts are statistically independent of those upon which the other units focus."

if you understand the basic principle behind GAN's and the min-max optimization game that the generator and the discriminator play, then in principle there is essentially no difference in GAN's and Schmidhuber's original paper(in principle) the principle is same and if Goodfellow didn't cite his paper then in academic terms it is called plagiarism and in real life, it is called stealing. There is essentially no debate here to have its either you accept that you took the idea or not. If you look at the original paper he proposes a similar min-max optimization game (This excerpt is directly taken from the original paper):

Maximization of the objective function(as defined in the original Schmidhuber's paper follows the following constraints) tends to force the representational units to take on binary values that maximize independence(referring to generators ability in GAN's to maximize it such that the outputs sampled from the vector of distribution encoded by the generator becomes impossible for the discriminator to discriminate) in addition to minimizing the reconstruction error(refers to minimizing the discriminator's loss function i.e maximizing its ability to differentiate between generated and real image).

Hence to anyone who understands this kind of stuff can realize who is at fault here and why Schmidhuber's annoyance is justified in the true sense. The statements in the bracket compare the principle similarity between GAN's and Schmidhuber's original 1991 paper.