Skip to main content

Questions tagged [actor-critic]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
0 answers
75 views

Batches in policy gradient methods – theory vs practice

I am currently trying to understand the implementation of batching in policy gradient / actor-critic methods. My understanding is that these methods in principal work as follows: collect a batch of $N$...
mathiasj's user avatar
1 vote
0 answers
24 views

Why PPO does not correct the value by the importance sampling ratio

I'm referring to this image present on this page : My question is, why we don't correct $R_t$ with the importance sampling ratio? At the end of the day, that return is sampled according to $\pi_{old}$...
Alberto's user avatar
  • 1,207
0 votes
0 answers
26 views

Actor-critic action-value update

I'm reading about Actor-Critic in lilianweng, and I'm wondering about why is this the action-value function update? what function are we taking the gradient of here?
ihadanny's user avatar
  • 3,340
0 votes
0 answers
35 views

Gaussian Log Probability [duplicate]

I have a code segment from this repo: https://github.com/toshikwa/slac.pytorch/blob/master/slac/utils.py I am reading a paper about Soft Actor-Critic, a reinforcement learning algorithm. They have ...
chadmc's user avatar
  • 1
0 votes
1 answer
452 views

actor update in DDPG algorithm (and in general actor-critic algorithms)

The update equations for the parameters of the actor and the critic are: $$ \delta_t = r_t+\gamma Q^\omega (x_{t+1},a_{t+1})-Q^\omega(x_t,a_t)$$ $$ \omega_{t+1} = \omega_t+\alpha_\omega \delta_t ...
Schach21's user avatar
  • 145
1 vote
1 answer
183 views

Deep deterministic policy gradient : which network do I have to use for testing?

We know that Deep deterministic policy gradient (henceforth ddpg) is characterized by two kind of neural networks: one related to the critic $Q$ the other to the actor $\mu$ with parameters $\theta^\...
Siderius's user avatar
1 vote
0 answers
67 views

Understanding the Neural Network Output using a Custom Gym Env continuous boundaries

I need help understanding some concepts on neural networks while using a Custom Gym Environment. ...
uBoscuBo's user avatar
2 votes
1 answer
633 views

How to define number of states in reinforcement learning

I'm a robotic engineer who's relatively new to reinforcement learning and I want to try to do simple reinforcement learning on a robot to optimize its velocity. I am however having trouble with ...
Mr_Melon's user avatar
2 votes
3 answers
445 views

Why the approximation of $\log \pi_{\theta}(a|s)$ improves numerical stability?

In Maxim Lapan's book Deep Reinforcement Learning Hands-on, section Continuous A2C, it says By definition, the probability density function of the Gaussian Distribution is $$f(x | \mu, \sigma^2) =...
jgauth's user avatar
  • 141
2 votes
0 answers
237 views

Can't use replay memory with policy gradient, why?

One of the approaches to improving the stability of the Policy Gradient family of methods is to use multiple environments in parallel. The reason behind this is the fundamental problem we ...
jgauth's user avatar
  • 141
4 votes
0 answers
571 views

A2C in TensorFlow 2 using model with two heads

I am implementing some of the basic reinforcement learning algorithms but ran into a problem with an online (one-step TD) A2C implementation where my reward seems to decrease over time instead of ...
Gregor's user avatar
  • 163
4 votes
1 answer
383 views

Advantage term in A2C

I am trying to understand A2C algorithm by an article Understanding Actor Critic Methods and A2C by Chris Yoon An author provides the following formula to compute an Advantage term: \begin{equation} ...
koryakinp's user avatar
  • 219
1 vote
0 answers
349 views

entropy coefficient in A3C [closed]

I see that my policy entropy is decreasing very fast and converges to zero in no time, causing the policy to sample the same action again and again (which results in a sub-optimal behavior). I think a ...
Gabizon's user avatar
  • 398
2 votes
0 answers
1k views

Reinforcement Learning: A2C agent does not learn

I am trying to implement an A2C algorithm, but for some reasons, my agent does not learn very well. I build a custom environment using Unity ML Agents. The environment is very simple: an agent can ...
koryakinp's user avatar
  • 219
1 vote
0 answers
67 views

Natural actor critic with nonlinear function approximation

I was wondering, if there was a way to implement natural actor critic without having to deal with choosing the right features for the action-state-space. Basically, in Sutton,Barto - Reinforcement ...
GreenLogic's user avatar
2 votes
1 answer
342 views

Understanding the temporal difference prediction error formula which uses a derivative

I'm very new to understanding the concept of prediction error underlying the output of the critic in the critic-actor method (RL learning), so bear with me, please. For the temporal difference ...
Hanna Haponenko's user avatar
6 votes
1 answer
2k views

Reinforcement Learning - What is the logic behind actor-critic methods? Why use a critic?

Following David Silver's course, I came across the actor-critic policy improvement algorithm family. It holds For one-step Markov decision processes that $$\nabla_{\theta}J(\theta) = \mathbb{E}_{\...
Gulzar's user avatar
  • 583
1 vote
1 answer
4k views

A2C Loss Function Explosion

I am training OpenAI's implementation of the A2C algorithm found here. Based on the mean episode reward graph below we can see it is in fact learning the policy function up until roughly 2000 updates:...
sccrthlt's user avatar
  • 160
3 votes
1 answer
1k views

What are the differences between contextual bandits, actor-citric methods, and continuous reinforcement learning?

Let's imagine we have a blackbox function f(X) -> y which we don't know. X is a vector of 10 continuous variables, which we ...
freesoul's user avatar
  • 141
13 votes
1 answer
15k views

Actor-critic loss function in reinforcement learning

In actor-critic learning for reinforcement learning, I understand you have an "actor" which is deciding the action to take, and a "critic" that then evaluates those actions, however, I'm confused on ...
tryingtolearn's user avatar