I was reading a paper Playing hard exploration games by watching YouTube https://arxiv.org/abs/1805.11592.

By my understanding the authors use convolutional neural networks to generate embeddings for the game state and the demonstration trajectory from a video recording (which is being learned from). They then use the embeddings for the game state and the video recording at some checkpoint to present an imitation reward to the learning agent (the only learning signal the agent receives). They use this signal to train a standard reinforcement learning agent.

However I do not understand what the authors are doing when they are predicting the Temporal Distance Intervals. Are they just predicting how different the embedding of the next frame should be? Or are they predicting what the agents next action should be?

I don't understand how predicting the Temporal Distance Intervals is related to generating the optimal policy for controlling the agent.

3.1 Temporal distance classification (TDC)

We first consider the unsupervised task of predicting the temporal distance ∆t between two frames of a single video sequence. This task requires an understanding of how visual features move and transform over time, thus encouraging an embedding that learns meaningful abstractions of environment dynamics conditioned on agent interactions. We cast this problem as a classification task, with K categories corresponding to temporal distance intervals,dk ∈{[0],1,[2],[3−4],[5−20],[21−200]}.Giventwoframesfromthesamevideo, v, w ∈ I, we learn to predict the interval dk s.t. ∆t ∈ dk. Specifically, we implement two functions: an embedding function φ : I → RN , and a classifier τtd : RN × RN → RK , both implemented as neural networks (see Section 5 for implementation details). We can then train τtd(φ(v), φ(w)) to predict the distribution over class labels, dk, using the following cross-entropy classification loss:

enter image description here

where yi and yˆi are the true and predicted label distributions respectively.


TDC + CMC are their methods of unsupervised representation learning.

Then RL is applied on inputs coming from the resulting embedding space, instead of the raw video space.

Having a high quality, compressed representation of the input space makes RL much more sample efficient.

| cite | improve this answer | |
  • $\begingroup$ So my current understating is that the classification of temporal distance intervals is just predicting the time interval between two frames (ie: 1-2 sec, 2-4 sec, ...). This forces the embeddings to learn a useful representation which is used to reward the imitation. Am I correct with my assumption that temporal distance intervals are just the time intervals? $\endgroup$ – Marko Arezina Jul 22 '18 at 1:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.