4
$\begingroup$

I am trying to understand A2C algorithm by an article Understanding Actor Critic Methods and A2C by Chris Yoon

An author provides the following formula to compute an Advantage term:

\begin{equation} A(s_t, a_t) = r_{t+1} + \gamma*V(s_{t+1}) - V(s_{t}) \end{equation}

I understand that formula as: an advantage of taking an action $a$ in a state $s$ equals to a reward received in the next state plus discounted value of a next state minus a value of a current state.

Let's say an episode consists of 5 states and discount factor $\gamma = 0.9$:

An array of rewards for each state: rewards = [10, 11, 10, 9, -12]

An array of values for each state: values = [7, 13, 14, 10, -11]

An array of done flags for each state: dones = [False, False, False, False, True]

\begin{equation} A(s_1, a_1) = 11 + 0.9*13 - 7 \\ A(s_2, a_2) = 10 + 0.9*14 - 13 \\ A(s_3, a_3) = 9 + 0.9*10 - 14 \\ A(s_4, a_4) = -12 + 0.9*(-11) - 10 \\ \end{equation}

First of all, is my calculation correct ? How to calculate an advantage of taking an action in a terminal state $A(s_5, a_5)$ ?

Update:

I populate rewards and values array in a following fashion:

rewards = []
values = []
dones = []
state = env.reset()
done = False

while not done:
    action, value = agent.get_action(state)
    state, reward, done = env.step(action)
    rewards.append(reward)
    values.append(value)
    dones.append(done)
$\endgroup$
0

1 Answer 1

2
$\begingroup$

Your calculations seem almost right, but your question about terminal state, and comparing your code+examples with this statement:

Let's say an episode consist of 5 states and discount factor $\gamma = 0.9$:

shows that you have a slight misunderstanding about how time step indices work. There are actually a couple of common conventions for this, as you can decide to associate the immediate reward with the same time step as the start state and action, or in the same time step as the response from the environment (when the next state is also returned). The formula you show uses the latter convention, and this does seem more common - e.g. Sutton & Barto's book uses it.

So, if an episode starts at time step $t=1$ in state $s_1$, takes an action, receives a reward and ends up in another state $s_2$, then takes another action and ends in terminal state, the full trajectory would look like this:

$$s_1, a_1, r_2, s_2, a_2, r_3, s_3$$

There are two actions $a_1$ and $a_2$, but no $a_3$ at the end (it is not possible to take an action in the terminal state). There are two rewards $r_2$ and $r_3$ but no $r_1$ (no reward is received simply for starting the environment). There are three states, including the terminal state $s_3$.

In your example, if you include the terminal state in your examples, you would have 6 states. You have not added it to your values array, because in your episode loop agent.get_action(state) is giving you $V(s)$ of the starting state, before taking each action. You never evaluate the terminal state.

You are missing the value of the terminal state. Tha value $V(s_T)$ of a terminal state is always, by definition, $0$. That is because the value is the expected sum of future rewards. In a terminal state there are by definition no future rewards, so the value is a sum over an empty list.

If you changed your example arrays so that the time indices aligned, you would have:

  • An array of rewards: rewards = [None, 10, 11, 10, 9, -12]

  • An array of values for each state: values = [7, 13, 14, 10, -11, 0]

Aside: In Python these are zero-indexed of course, but in the notation I am - like you - showing the first element as indexed by $t=1$

So, your equations are not quite right. Here is what the first one should be:

$$A(s_1, a_1) = 10 + 0.9*13 - 7$$

. . because it should be using $r_2$ for the immediate reward, but you had jumped to $r_3$ due to the misalignment of your data.

This also partly answers your question

How to calculate an advantage of taking an action in a terminal state $A(s_5, a_5)$ ?

In the actual terminal state, any value function will be $0$ by definition, and you cannot take an action in that state anyway. However, if you need the value for backing up to previous values, you can use $A(s_T,\cdot) = 0$. In your case, $T=6$.

You can calculate $A(s_5, a_5)$ in your case because $s_5$ is not the terminal state:

$$A(s_5, a_5) = -12 + 0.9*0 -(-11)$$

$\endgroup$
1
  • $\begingroup$ thank you sir! That is a great explanation! $\endgroup$
    – koryakinp
    Oct 25, 2019 at 17:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.