1
$\begingroup$

In the A3C algorithm from the original paper:

enter image description here

the gradient with respect to log policy involves the term

$$\log \pi(a_i|s_i;\theta')$$

where $s_i$ is the state of the environment at time step $i$, and $a_i$ is the action produced by the policy. If I understand correctly, the output of the policy is a softmax function, so that if there are $n$ different actions, then we get the $n$-dimensional vector output

$$\pi(s_i;\theta')=\left(\frac{e^{o_1(s_i)}}{\sum_{l=1}^n e^{o_l(s_i)}},\frac{e^{o_2(s_i)}}{\sum_{l=1}^n e^{o_l(s_i)}},...,\frac{e^{o_n(s_i)}}{\sum_{l=1}^n e^{o_l(s_i)}}\right),$$

where the $o_j(s)$ are softmax layer activations obtained from forward propagation of state $s_i$ through the neural network.

Do I understand correctly that in the A3C algorithm above the term $\log \pi(a_i|s_i;\theta')$ refers to

$$\log \pi(a_i|s_i;\theta') = \log\left(\frac{e^{o_j(s_i)}}{\sum_{l=1}^n e^{o_l(s_i)}}\right)$$

with index $j$ referring to the position of the largest element in vector $\pi(s_i;\theta')$ above? Or maybe all action options should be contributing according to their probabilistic weights, like so

$$\log \pi(a_i|s_i;\theta') = \sum_{j=1}^n\log\left(\frac{e^{o_j(s_i)}}{\sum_{l=1}^n e^{o_l(s_i)}}\right)~~~?$$

Or perhaps neither of these expressions is correct? In that case, what is the correct explicit formula for the expression $\log \pi(a_i|s_i;\theta')$ in terms of softmax layer activations $o_j$?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.