Generative adversarial networks application to reinforcement learning Can generative adversarial networks be applied to reinforcement learning? 
Like imitation learning or inverse reinforcement learning?
 A: 
Can generative adversarial networks be applied to reinforcement learning?

Yes.
How the training of a GAN is formulated has been shown as applicable to inverse RL. We know a GAN has a Discriminator $D$ whose objective is to distiguish between the real data and fake data created by the Generator. The Generator $G$, on the other hand, learns to generate samples that look as similar as possible to the real data.
How this applies to Imitation and Inverse RL
The GAN Discriminator learns by reducing the Binary Cross-Entropy Loss(BCE) between the real and fake data:
$log(D_{\phi}(x))+log(1−D_{\phi}(G(z)))$, where $x$ is a real sample, and $G(z)$ is a fake output from the Generator.
Similar to this, Inverse and Imitation RL use expert demonstrations to ultimately train a policy. These demonstrations can be regarded as the real data while the transitions the learning policy $\pi_{\theta}$ collects when running in the environment, as the fake data. We utilize the Discriminator in this case to maximize the probability of correctly classifying a transition $\tau$ as being from expert demonstrations $*p$ or a sample from the learned policy $\pi_{\theta}$. We can express this as:

The transitions $\tau$ are drawn from the expert distribution $\tau \sim *p$ in the first part of the equation, and drawn from the policy samples $\tau \sim \pi_{\theta}$ in the second part. In other words, we try to maximize the likelihood of a transition coming from the expert distribution $*p$ while minimizing that from the policy distribution $\pi_{\theta}$.
This improves the objective of the Discriminator which is to minimize BCE loss between expert demos and generated samples.
How does this meet the RL objective?
The objective of RL is to maximize the cumulative future reward $r(s_t, a_t)$. Since imitation and inverse RL do not utilize an inbuilt reward function, we utilize the Discriminator objective to replace the policy optimization objective.
$r_{\phi}(s_t, a_t) = log(D_{\sim *p}(s, a)) − log(1 − D_{\sim \pi_{\theta}} (s, a))$
So we turn the Discriminator minimization objective into maximization.
The learned policy is then updated using $r_{\phi}$ to make it's samples harder to distinguish from the expert demos - much like a Generator in a GAN:
$\sum \nabla_{\theta} \pi_{\theta} r_{\phi}(s_t, a_t)$
We can summarize all this in an image:

And where is the difference between Imitation and Inverse RL here?
Quick definitions:

*

*Imitation Learning: Recovers a policy from the expert demonstrations by mimicking the expert behaviour.

*Inverse RL: Infers the expert's intention by recovering a reward function from the demonstrations. This means that when the expert makes mistakes or shows sub-optimal behaviour, the policy won't overfit to such behaviour/mistakes but instead focus on the expert's intention.

The difference is in how we formulate the Discriminator. For imitation learning, the Discriminator $D$ will be a standard binary neural net classifier that can't optimize for the expert's intention - it just learns to mimic what the expert does.
In inverse RL though, $D$ will take the form of a learnable reward function $f_{\phi}(\tau)$, expressed as:
$D_{\phi} = \frac{exp f_{\phi}(\tau)}{exp f_{\phi}(\tau) + \pi_{\theta}(a|s)}$
Updating the Discriminator $D_{\phi}$ updates the reward function $f_{\phi}$.
In other areas, GANs have also been used for Transfer Learning in RL through Domain Adaptation during Sim-to-Real transfer. Here's an example. This uses a complete GAN architecture with both a Discriminator and a Generator.
