# Meaning of “differentiable model of the environment dynamics” in RL

Does anyone know what the term "differentiable model of the environment dynamics" means exactly? I read the paper Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments where it is stated for example on page 4: "we do not assume a differentiable model of the environment dynamics, unlike in [24] ...". Even after reading [24] it is not clear to me. Does it mean that the information about the environment (which is incorporated in the state $\mathcal{S}$) changes with the agents actions $\mathcal{A}$?

The dynamics of an environment describes how the state changes when you take an action. In the context of the deterministic environment in [24], a dynamics model would learn a mapping $\hat s' = f(s,a)$ which predicts the next state $s'$ resulting from taking the action $a$ in state $s$. A differentiable model simply means that this mapping is differentiable.
• Just one minor question: differentiable in what? in "$s$", "$a$", or "$s,a$"? – Tik0 Jul 22 '18 at 15:21
• both $s$ and $a$ – shimao Jul 22 '18 at 15:22