Questions tagged [actor-critic]
The actor-critic tag has no usage guidance.
11
questions
0
votes
0answers
12 views
entropy coefficient in A3C
I see that my policy entropy is decreasing very fast and converges to zero in no time, causing the policy to sample the same action again and again (which results in a sub-optimal behavior).
I think a ...
2
votes
0answers
73 views
Reinforcement Learning: A2C agent does not learn
I am trying to implement an A2C algorithm, but for some reasons, my agent does not learn very well.
I build a custom environment using Unity ML Agents. The environment is very simple: an agent can ...
0
votes
0answers
12 views
Natural actor critic with nonlinear function approximation
I was wondering, if there was a way to implement natural actor critic without having to deal with choosing the right features for the action-state-space.
Basically, in Sutton,Barto - Reinforcement ...
0
votes
0answers
20 views
Is it okay to calculate all the gradients for an LSTM at once?
I'm trying to use the AC method reported in "Asynchronous Methods for Deep Reinforcement Learning" for a project. The relevant algorithm is shown in pseudocode at the bottom, Algorithm S3.
I'm using ...
0
votes
0answers
71 views
Monte Carlo actor critic algorithm
Has anyone implemented Monte Carlo on policy actor-critic algorithm or know a codebase online and can share it?
P.S: I am not sure if this is the right place to post this, sorry for that.
2
votes
1answer
108 views
Understanding the temporal difference prediction error formula which uses a derivative
I'm very new to understanding the concept of prediction error underlying the output of the critic in the critic-actor method (RL learning), so bear with me, please.
For the temporal difference ...
3
votes
1answer
234 views
Reinforcement Learning - What is the logic behind actor-critic methods? Why use a critic?
Following David Silver's course, I came across the actor-critic policy improvement algorithm family.
It holds For one-step Markov decision processes that
$$\nabla_{\theta}J(\theta) = \mathbb{E}_{\...
1
vote
0answers
66 views
High variance of returns using Asynchronous Actor-Critic Agents (A3C) on CartPole [closed]
I ran the code from the Tensorflow blog with modified running average function (it takes a running mean of the last 3 episodes only) and notice strange behavior. Although the model achieves episode ...
1
vote
1answer
1k views
A2C Loss Function Explosion
I am training OpenAI's implementation of the A2C algorithm found here. Based on the mean episode reward graph below we can see it is in fact learning the policy function up until roughly 2000 updates:...
1
vote
1answer
547 views
What are the differences between contextual bandits, actor-citric methods, and continuous reinforcement learning?
Let's imagine we have a blackbox function f(X) -> y which we don't know. X is a vector of 10 continuous variables, which we ...
7
votes
1answer
3k views
Actor-critic loss function in reinforcement learning
In actor-critic learning for reinforcement learning, I understand you have an "actor" which is deciding the action to take, and a "critic" that then evaluates those actions, however, I'm confused on ...
