Questions tagged [actor-critic]
The actor-critic tag has no usage guidance.
15
questions
2
votes
1answer
30 views
How to define number of states in reinforcement learning
I'm a robotic engineer who's relatively new to reinforcement learning and I want to try to do simple reinforcement learning on a robot to optimize its velocity. I am however having trouble with ...
2
votes
3answers
30 views
Why the approximation of $\log \pi_{\theta}(a|s)$ improves numerical stability?
In Maxim Lapan's book Deep Reinforcement Learning Hands-on, section Continuous A2C, it says
By definition, the probability density function of the Gaussian
Distribution is $$f(x | \mu, \sigma^2) =...
0
votes
0answers
17 views
Instability in Multi-Agent Reinforcement Learning
I've got a program tackling a Multi-Agent Markov Decision Process using Deep Reinforcement Learning. To solve the problem I've tried a few algorithms broadly belonging to these two classes:
A2C (...
2
votes
0answers
13 views
Can't use replay memory with policy gradient, why?
One of the approaches to improving the stability of the Policy
Gradient family of methods is to use multiple environments in
parallel. The reason behind this is the fundamental problem we
...
2
votes
0answers
142 views
A2C in TensorFlow 2 using model with two heads
I am implementing some of the basic reinforcement learning algorithms but ran into a problem with an online (one-step TD) A2C implementation where my reward seems to decrease over time instead of ...
0
votes
0answers
45 views
Parameters of Multi-agent actor-critic MADDPG algorithm
I am reading the original paper of MADDPG. The thing I cannot understand, however, is that why they use the same parameters for both Actor and Critic. See below equation (4), (5 - updating actor), (6 -...
2
votes
1answer
77 views
Advantage term in A2C
I am trying to understand A2C algorithm by an article Understanding Actor Critic Methods and A2C by Chris Yoon
An author provides the following formula to compute an Advantage term:
\begin{equation} ...
1
vote
0answers
117 views
entropy coefficient in A3C [closed]
I see that my policy entropy is decreasing very fast and converges to zero in no time, causing the policy to sample the same action again and again (which results in a sub-optimal behavior).
I think a ...
2
votes
0answers
240 views
Reinforcement Learning: A2C agent does not learn
I am trying to implement an A2C algorithm, but for some reasons, my agent does not learn very well.
I build a custom environment using Unity ML Agents. The environment is very simple: an agent can ...
1
vote
0answers
26 views
Natural actor critic with nonlinear function approximation
I was wondering, if there was a way to implement natural actor critic without having to deal with choosing the right features for the action-state-space.
Basically, in Sutton,Barto - Reinforcement ...
2
votes
1answer
173 views
Understanding the temporal difference prediction error formula which uses a derivative
I'm very new to understanding the concept of prediction error underlying the output of the critic in the critic-actor method (RL learning), so bear with me, please.
For the temporal difference ...
4
votes
1answer
581 views
Reinforcement Learning - What is the logic behind actor-critic methods? Why use a critic?
Following David Silver's course, I came across the actor-critic policy improvement algorithm family.
It holds For one-step Markov decision processes that
$$\nabla_{\theta}J(\theta) = \mathbb{E}_{\...
1
vote
1answer
2k views
A2C Loss Function Explosion
I am training OpenAI's implementation of the A2C algorithm found here. Based on the mean episode reward graph below we can see it is in fact learning the policy function up until roughly 2000 updates:...
1
vote
1answer
695 views
What are the differences between contextual bandits, actor-citric methods, and continuous reinforcement learning?
Let's imagine we have a blackbox function f(X) -> y which we don't know. X is a vector of 10 continuous variables, which we ...
9
votes
1answer
4k views
Actor-critic loss function in reinforcement learning
In actor-critic learning for reinforcement learning, I understand you have an "actor" which is deciding the action to take, and a "critic" that then evaluates those actions, however, I'm confused on ...
