3
$\begingroup$

I have seen in imitation learning papers that they have mentioned their algorithm suffered from "DAgger problem". Can someone please explain me this.

Is it about taking the same action by the agent again and again?

5. Conclusion
Although we have implemented a model that enables InfoGAIL to use visual input and intended to show that it increases the applicability of InfoGAIL, we instead discovered that InfoGAIL (and by extension GAIL) suffer from the “DAgger problem.” This is not through a fault of the original algorithms, but it is a fact of using distribution matching (like with a generative adversarial framework) with experts that take only one type of action or with environments that lend themselves to these kinds of skewed state-action distributions.

Source:

  • Ivanovic, et al. Learning a Visual State Representation for Generative Adversarial Imitation Learning (pdf)
$\endgroup$
0

1 Answer 1

4
$\begingroup$

(Note that I don't have much expertise with this material.)

This seems to be a term they coined themselves in the same paper (my italics):

4.2. Model Generalization
In order to show that our image-based model generalizes to states that it has not seen before (even states that the expert has not seen before) we devised the following experiment on Game 1: Navigate to Object (“Go to Goal0”).
...
Very surprisingly, we do not see generalization at all. We actually see the agent taking more-or-less random actions in the unexplored states. We believe that this occurs not because of a fault in the GAIL or InfoGAIL algorithms, but because this “Grid World” environment and expert lend themselves to creating sharply-peaked state-action distributions. An example of how this could occur is shown in Fig. 5. Since the agent learns to match this state-action distribution, it will similarly always take actions to the right of the world. However, if it overshoots the goal (due to some noise in the agent’s output action), the agent will never make it back to the goal since its learned state-action distribution is completely skewed to taking actions to the right (to match the expert). Thus, it will never take actions to the left to correct its overshoot. We call this problem the “DAgger problem” as it is reminiscent of a previous RL method [15] that suffered from the inability to generalize because it cannot work on states that an expert has not been to.
...
[15] S. Ross, G. J. Gordon, and J. A. Bagnell. No-regret reductions for imitation learning and structured prediction. CoRR, abs/1011.0686, 2010.

I can't seem to find a copy of Ross et al. online, but the description seems clear enough to me. The algorithm learns to match an expert that has never been in a novel situation and the actions it has learned to mimic mindlessly1 are inappropriate for the new situation.

1. This is editorializing on my part, but I think it's relevant here to remind ourselves here that strong AI doesn't exist yet and that's part of the explanation for why this fails in the way that it does.

$\endgroup$
2
  • $\begingroup$ yeah I also tried to find something on this but failed. You are correct $\endgroup$ Jun 27, 2018 at 23:49
  • 2
    $\begingroup$ I believe the paper they're referring to is "A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning" (this is the paper that introduces the DAgger algorithm), which is freely available online. The problem that DAgger is intended to solve (which is what they're calling the "DAgger problem") is essentially what you said, that the distribution of states the expert encounters doesn't cover all the states the learned agent encounters. $\endgroup$
    – amiller27
    Sep 7, 2018 at 3:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.