I self-published the basic idea of a deterministic variety of generative adversarial networks (GANs) in a 2010 blog post (archive.org). I had searched for but could not find anything similar anywhere, and had no time to try implementing it. I was not and still am not a neural network researcher and have no connections in the field. I'll copy-paste the blog post here:
A method for training artificial neural networks to generate missing
data within a variable context. As the idea is hard to put in a single
sentence, I will use an example:
An image may have missing pixels (let's say, under a smudge). How can
one restore the missing pixels, knowing only the surrounding pixels?
One approach would be a "generator" neural network that, given the
surrounding pixels as input, generates the missing pixels.
But how to train such a network? One can't expect the network to
exactly produce the missing pixels. Imagine, for example, that the
missing data is a patch of grass. One could teach the network with a
bunch of images of lawns, with portions removed. The teacher knows the
data that is missing, and could score the network according to the
root mean square difference (RMSD) between the generated patch of
grass and the original data. The problem is that if the generator
encounters an image that is not part of the training set, it would be
impossible for the neural network to put all the leaves, especially in
the middle of the patch, in exactly the right places. The lowest RMSD
error would probably be achieved by the network filling the middle
area of the patch with a solid color that is the average of the color
of pixels in typical images of grass. If the network tried to generate
grass that looks convincing to a human and as such fulfills its
purpose, there would be an unfortunate penalty by the RMSD metric.
My idea is this (see figure below): Train simultaneously with the
generator a classifier network that is given, in random or alternating
sequence, generated and original data. The classifier then has to
guess, in the context of the surrounding image context, whether the
input is original (1) or generated (0). The generator network is
simultaneously trying to get a high score (1) from the classifier. The
outcome, hopefully, is that both networks start out really simple, and
progress towards generating and recognizing more and more advanced
features, approaching and possibly defeating human's ability to
discern between the generated data and the original. If multiple
training samples are considered for each score, then RMSD is the
correct error metric to use, as this will encourage the classifier
network to output probabilities.
Artificial neural network training setup
When I mention RMSD at the end I mean the error metric for the "probability estimate", not the pixel values.
I originally started considering the use of neural networks in 2000 (comp.dsp post) to generate missing high frequencies for up-sampled (resampled to a higher sampling frequency) digital audio, in a way that would be convincing rather than accurate. In 2001 I collected an audio library for the training. Here are parts of an EFNet #musicdsp Internet Relay Chat (IRC) log from 20 January 2006 in which I (yehar) talk about the idea with another user (_Beta):
[22:18] <yehar> the problem with samples is that if you don't have something "up there" already then what can you do if you upsample...
[22:22] <yehar> i once collected a big library of sounds so that i could develop a "smart" algo to solve this exact problem
[22:22] <yehar> i would have used neural networks
[22:22] <yehar> but i didn't finish the job :-D
[22:23] <_Beta> problem with neural networks is that you have to have some way of measuring the goodness of results
[22:24] <yehar> beta: i have this idea that you can develop a "listener" at the same time as you develop the "smart up-there sound creator"
[22:26] <yehar> beta: and this listener will learn to detect when it's listening a created or a natural up-there spectrum. and the creator develops at the same time to try to circumvent this detection
Sometime between 2006 and 2010, a friend invited an expert to take a look at my idea and discuss it with me. They thought that it was interesting, but said that it wasn't cost-effective to train two networks when a single network can do the job. I was never sure if they did not get the core idea or if they immediately saw a way to formulate it as a single network, perhaps with a bottleneck somewhere in the topology to separate it into two parts. This was at a time when I didn't even know that backpropagation is still the de-facto training method (learned that making videos in the Deep Dream craze of 2015). Over the years I had talked about my idea with a couple of data scientists and others that I thought might be interested, but the response was mild.
In May 2017 I saw Ian Goodfellow's tutorial presentation on YouTube [Mirror], which totally made my day. It appeared to me as the same basic idea, with differences as I currently understand outlined below, and the hard work had been done to make it give good results. Also he gave a theory, or based everything on a theory, of why it should work, while I never did any sort of a formal analysis of my idea. Goodfellow's presentation answered questions that I had had and much more.
Goodfellow's GAN and his suggested extensions include a noise source in the generator. I never thought of including a noise source but have instead the training data context, better matching the idea to a conditional GAN (cGAN) without a noise vector input and with the model conditioned on a part of the data. My current understanding based on Mathieu et al. 2016 is that a noise source is not needed for useful results if there is enough input variability. The other difference is that Goodfellow's GAN minimizes log-likelihood. Later, a least squares GAN (LSGAN) has been introduced (Mao et al. 2017) which matches my RMSD suggestion. So, my idea would match that of a conditional least squares generative adversarial network (cLSGAN) without a noise vector input to the generator and with a part of the data as the conditioning input. A generative generator samples from an approximation of the data distribution. I do now know if and doubt that real-world noisy input would enable that with my idea, but that is not to say that the results would not be useful if it didn't.
The differences mentioned in the above are the primary reason why I believe Goodfellow did not know or hear about my idea. Another is that my blog has had no other machine learning content, so it would have enjoyed very limited exposure in machine learning circles.
It is a conflict of interests when a reviewer puts pressure on an author to cite the reviewer's own work.