Does anyone know what the term "differentiable model of the environment dynamics" means exactly? I read the paper Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments where it is stated for example on page 4: "we do not assume a differentiable model of the environment dynamics, unlike in [24] ...". Even after reading [24] it is not clear to me. Does it mean that the information about the environment (which is incorporated in the state $\mathcal{S}$) changes with the agents actions $\mathcal{A}$?


The dynamics of an environment describes how the state changes when you take an action. In the context of the deterministic environment in [24], a dynamics model would learn a mapping $\hat s' = f(s,a)$ which predicts the next state $s'$ resulting from taking the action $a$ in state $s$. A differentiable model simply means that this mapping is differentiable.

Also, in [24] they didn't actually use a differentiable model, they had direct access to the dynamics of the environment itself, and the dynamics of the environment were differentiable!

  • $\begingroup$ Just one minor question: differentiable in what? in "$s$", "$a$", or "$s,a$"? $\endgroup$ – Tik0 Jul 22 '18 at 15:21
  • 1
    $\begingroup$ both $s$ and $a$ $\endgroup$ – shimao Jul 22 '18 at 15:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.