Questions tagged [actor-critic]
The actor-critic tag has no usage guidance.
15
questions
0
votes
0answers
14 views
Why is my actor loss reversed?
I'm trying to train RL model with REINFORCE and a Critic network.
The agent is learning, but I don't understand the look of my actor loss.
Here are the curves:
In this task the goal is to minimize ...
2
votes
1answer
104 views
How to define number of states in reinforcement learning
I'm a robotic engineer who's relatively new to reinforcement learning and I want to try to do simple reinforcement learning on a robot to optimize its velocity. I am however having trouble with ...
2
votes
3answers
36 views
Why the approximation of $\log \pi_{\theta}(a|s)$ improves numerical stability?
In Maxim Lapan's book Deep Reinforcement Learning Hands-on, section Continuous A2C, it says
By definition, the probability density function of the Gaussian
Distribution is $$f(x | \mu, \sigma^2) =...
0
votes
0answers
24 views
Instability in Multi-Agent Reinforcement Learning
I've got a program tackling a Multi-Agent Markov Decision Process using Deep Reinforcement Learning. To solve the problem I've tried a few algorithms broadly belonging to these two classes:
A2C (...
2
votes
0answers
55 views
Can't use replay memory with policy gradient, why?
One of the approaches to improving the stability of the Policy
Gradient family of methods is to use multiple environments in
parallel. The reason behind this is the fundamental problem we
...
2
votes
0answers
289 views
A2C in TensorFlow 2 using model with two heads
I am implementing some of the basic reinforcement learning algorithms but ran into a problem with an online (one-step TD) A2C implementation where my reward seems to decrease over time instead of ...
3
votes
1answer
151 views
Advantage term in A2C
I am trying to understand A2C algorithm by an article Understanding Actor Critic Methods and A2C by Chris Yoon
An author provides the following formula to compute an Advantage term:
\begin{equation} ...
1
vote
0answers
202 views
entropy coefficient in A3C [closed]
I see that my policy entropy is decreasing very fast and converges to zero in no time, causing the policy to sample the same action again and again (which results in a sub-optimal behavior).
I think a ...
2
votes
0answers
381 views
Reinforcement Learning: A2C agent does not learn
I am trying to implement an A2C algorithm, but for some reasons, my agent does not learn very well.
I build a custom environment using Unity ML Agents. The environment is very simple: an agent can ...
1
vote
0answers
30 views
Natural actor critic with nonlinear function approximation
I was wondering, if there was a way to implement natural actor critic without having to deal with choosing the right features for the action-state-space.
Basically, in Sutton,Barto - Reinforcement ...
2
votes
1answer
239 views
Understanding the temporal difference prediction error formula which uses a derivative
I'm very new to understanding the concept of prediction error underlying the output of the critic in the critic-actor method (RL learning), so bear with me, please.
For the temporal difference ...
4
votes
1answer
857 views
Reinforcement Learning - What is the logic behind actor-critic methods? Why use a critic?
Following David Silver's course, I came across the actor-critic policy improvement algorithm family.
It holds For one-step Markov decision processes that
$$\nabla_{\theta}J(\theta) = \mathbb{E}_{\...
1
vote
1answer
3k views
A2C Loss Function Explosion
I am training OpenAI's implementation of the A2C algorithm found here. Based on the mean episode reward graph below we can see it is in fact learning the policy function up until roughly 2000 updates:...
2
votes
1answer
821 views
What are the differences between contextual bandits, actor-citric methods, and continuous reinforcement learning?
Let's imagine we have a blackbox function f(X) -> y which we don't know. X is a vector of 10 continuous variables, which we ...
10
votes
1answer
6k views
Actor-critic loss function in reinforcement learning
In actor-critic learning for reinforcement learning, I understand you have an "actor" which is deciding the action to take, and a "critic" that then evaluates those actions, however, I'm confused on ...
