I am studying deterministic actor-critic algorithms in reinforcement learning.

I try to give a brief explanation of actor-critic algorithms before jumping into the mathematics. The actor takes in state $s$ and outputs a deterministic action $a$ based on the distribution policy $u$.

The state and action are fed into the critic. The critic sees how good it is to take a particular action from a given state using the action-value function $Q(s,a,w)$.

The critic is then updated via temporal difference (TD) learning and the actor updated in the direction of the critic

Thus it can be seen that the actor's goal is to try and maximise the state action value function $Q(s,a,w)$ by picking the best actions in the given state.

enter image description here

I am having trouble understanding the mathematics behind the updating of the actor.

The below equation gives how the actor is updated.

\begin{equation} \frac{\partial l}{\partial u} = \frac{\partial Q(s, a, w)}{\partial a} \frac{\partial a}{\partial u} \end{equation}

What I understand is that we are taking the partial derivative of $l$ with respect to $u$, and we are backpropogating the critic gradient to the actor.

It seems that $l$ is a differentiable function of the variable $a$, but I am confused when it comes to describing what is happening in the equation above as it seems to consist of two functions multiplied together.

Can someone kindly explain what is really happening in the mathematics above?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.