I read on https://en.wikipedia.org/wiki/Generative_adversarial_networks :

[Generative adversarial networks] were introduced by Ian Goodfellow et al in 2014.

but Jurgen Schmidhuber claims to have performed similar work earlier in that direction (e.g., there was some debate at NIPS 2016 during the generative adversarial networks tutorial: https://channel9.msdn.com/Events/Neural-Information-Processing-Systems-Conference/Neural-Information-Processing-Systems-Conference-NIPS-2016/Generative-Adversarial-Networks see 1h03min).

Was the idea behind generative adversarial networks first publicly introduced by Jürgen Schmidhuber? If not, how similar were Jürgen Schmidhuber's ideas?

I self-published the basic idea of generative adversarial networks (GANs) in a 2010 blog post (archive.org). I had searched for but could not find anything similar anywhere, and had no time to try implementing it. I was not and still am not a neural network researcher and have no connections in the field. I'll copy-paste the blog post here:

2010-02-24

A method for training artificial neural networks to generate missing data within a variable context. As the idea is hard to put in a single sentence, I will use an example:

An image may have missing pixels (let's say, under a smudge). How can one restore the missing pixels, knowing only the surrounding pixels? One approach would be a "generator" neural network that, given the surrounding pixels as input, generates the missing pixels.

But how to train such a network? One can't expect the network to exactly produce the missing pixels. Imagine, for example, that the missing data is a patch of grass. One could teach the network with a bunch of images of lawns, with portions removed. The teacher knows the data that is missing, and could score the network according to the root mean square difference (RMSD) between the generated patch of grass and the original data. The problem is that if the generator encounters an image that is not part of the training set, it would be impossible for the neural network to put all the leaves, especially in the middle of the patch, in exactly the right places. The lowest RMSD error would probably be achieved by the network filling the middle area of the patch with a solid color that is the average of the color of pixels in typical images of grass. If the network tried to generate grass that looks convincing to a human and as such fulfills its purpose, there would be an unfortunate penalty by the RMSD metric.

My idea is this (see figure below): Train simultaneously with the generator a classifier network that is given, in random or alternating sequence, generated and original data. The classifier then has to guess, in the context of the surrounding image context, whether the input is original (1) or generated (0). The generator network is simultaneously trying to get a high score (1) from the classifier. The outcome, hopefully, is that both networks start out really simple, and progress towards generating and recognizing more and more advanced features, approaching and possibly defeating human's ability to discern between the generated data and the original. If multiple training samples are considered for each score, then RMSD is the correct error metric to use, as this will encourage the classifier network to output probabilities.

enter image description here
Artificial neural network training setup

I originally started considering the use of neural networks in 2000 (comp.dsp post) to generate missing high frequencies for up-sampled (resampled to a higher sampling frequency) digital audio, in a way that would be convincing rather than accurate. In 2001 I collected an audio library for the training. Here are parts of an EFNet #musicdsp Internet Relay Chat (IRC) log from 20 January 2006 in which I (yehar) talk about the idea with another user (_Beta):

[22:18] <yehar> the problem with samples is that if you don't have something "up there" already then what can you do if you upsample...
[22:22] <yehar> i once collected a big library of sounds so that i could develop a "smart" algo to solve this exact problem
[22:22] <yehar> i would have used neural networks
[22:22] <yehar> but i didn't finish the job :-D
[22:23] <_Beta> problem with neural networks is that you have to have some way of measuring the goodness of results
[22:24] <yehar> beta: i have this idea that you can develop a "listener" at the same time as you develop the "smart up-there sound creator"
[22:26] <yehar> beta: and this listener will learn to detect when it's listening a created or a natural up-there spectrum. and the creator develops at the same time to try to circumvent this detection

Sometime between 2006 and 2010, a friend invited an expert to take a look at my idea and discuss it with me. They thought that it was interesting, but said that it wasn't cost-effective to train two networks when a single network can do the job. I was never sure if they did not get the core idea or if they immediately saw a way to formulate it as a single network, perhaps with a bottleneck somewhere in the topology to separate it into two parts. This was at a time when I didn't even know that backpropagation is still the de-facto training method (learned that making videos in the Deep Dream craze of 2015). Over the years I had talked about my idea with a couple of data scientists and others that I thought might be interested, but the response was mild.

In May 2017 I saw Ian Goodfellow's tutorial presentation on YouTube [Mirror], which totally made my day. It was the same basic idea, and the hard work had been done to make it give good results. The presentation answered questions that I had had and much more.

It is a conflict of interests when a reviewer puts pressure on an author to cite the reviewer's own work.

  • 4
    I must say, I am hugely impressed. It appears that your blog post deserves credit for the idea as well, even though Ian and team may have discovered it independently. – user2808118 Nov 13 '17 at 14:21
  • Had you published your work, you might have become the person that inspired the adversarial revolution. GASP!! – user2808118 May 21 at 11:00
  • @user2808118 That would've required working on it, to produce some results to publish, not just thinking about it. – Olli Niemitalo May 21 at 16:14

An answer from Ian Goodfellow on Was Jürgen Schmidhuber right when he claimed credit for GANs at NIPS 2016? posted on 2017-03-21:

He isn’t claiming credit for GANs, exactly. It’s more complicated.

You can see what he wrote in his own words when he was a reviewer of the NIPS 2014 submission on GANs: Export Reviews, Discussions, Author Feedback and Meta-Reviews (mirror)

He’s the reviewer that asked us to change the name of GANs to “inverse PM.”

Here's the paper he believes is not being sufficiently acknowledged: http://ftp://ftp.idsia.ch/pub/juergen/factorial.pdf (mirror)

I don’t like that there is no good way to have issues like this adjudicated. I contacted the NIPS organizers and asked if there is a way for Jürgen to file a complaint about me and have a committee of NIPS representatives judge whether my publication treats his unfairly. They said there is no such process available.

I personally don’t think that there is any significant connection between predictability minimization and GANs. I have never had any problem acknowledging connections between GANs and other algorithms that actually are related, like noise-contrastive estimation and self-supervised boosting.

Jürgen and I intend to write a paper together soon describing the similarities and differences between PM and GANs, assuming we’re able to agree on what those are.

  • 1
    Unfortunately there doesn't seem to be a paper between the two forthcoming. :-( – ComputerScientist Jul 14 '17 at 1:14
  • @ComputerScientist yep still waiting, someone else could take care of it otherwise :-) – Franck Dernoncourt Jul 14 '17 at 1:50

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.