I'm looking for academic papers or other credible sources focusing on the topic of parralelized reinforcement learning, specifically Q-learning. I'm mostly interested in methods of sharing Q-table between processes (or joining/syncing them together if each process have it's own). I'd also appreciate a brief description of method used in linked/mentioned sources.

My question is how to parallelize Q-learning which uses neural network as Q-table approximation. I'm looking for credible sources.

I should mention that I use neural network (PyBrain) as approximation.

  • Start with IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures. There is plenty of references for older work. The idea is to do smart batching with off-policyness correction so that we can get a boost from using GPUs. In general, the idea is in sharing experiences, not the gradients between actors and learners.
  • Look also at A3C/A2C which does async/sync updates on the gradients the master node gets from the workers.
  • PPO also does multi-thread learning when workers share the collected experience with the master node which updates the weights.
  • Finally, since you asked for DQN, GORILA paper which separates actors, learners and the main node which stores the weights (in tensorflow setting it's called parameter server). As far as I get it, this paper inspired IMPALA.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.