# Confusions about how to choose an action in deterministic policy gradient

I read "Continuous Control with Deep Reinforcement Learning". it says

The DPG algorithm maintains a parameterized actor function μ(s|θμ) which specifies the current policy by deterministically mapping states to a specific action.

I thought DPG simply chose the action with maximum $\mu(s|\theta_\mu)$(with noise for exploration). But when I read "Deterministic Policy Gradient Algorithms", it seems to say a different story:

To ensure that our deterministic policy gradient algorithms continue to explore satisfactorily, we introduce an off-policy learning algorithm. The basic idea is to choose actions according to a stochastic behavior policy (to ensure adequate exploration), but to learn about a deterministic target policy (exploiting the efficiency of the deterministic policy gradient)

According to my understanding, it says to choose an action as the stochastic policy gradient, but to learn the policy parameters using $\nabla_\theta Q(s,a)$

I'm not sure whether I've misunderstood something, but now I get confused -- how does DPG select an action exactly? or both above cases work fine?

• Are you asking how DPG selects an action after you have trained it, or while training it? These are different. Commented Apr 21, 2018 at 9:17
• @NeilSlater Sorry, I'm not so sure what's the difference. Could you please explain it more detailed? If you mean exploration happens only during the training but not after the training, then I was asking how DPG selected an action during the training Commented Apr 22, 2018 at 3:04

The function $\mu(s,\theta_{\mu})$ is not a value function, and you cannot take a meaningful maximum over it when deciding which action to take.

In fact the function must return a single action, that's what the Deterministic part of Deterministic Policy Gradient means. In addition the function $\mu(s,\theta_{\mu})$ must be differentiable with respect to $\theta$. So DPG is most easily applied in continuous action problems, where the action is a vector of real values, such as amount of steering and acceleration for a robotic vehicle.

How does DPG select an action exactly?

In order to explore, DPG selects an action by adding an exploring function (noted as $\mathcal{N}$) to the deterministic policy, creating $\mu'(s)$ as the behaviour policy:

$$\mu'(s_t) = \mu(s_t,\theta^{\mu}_t) + \mathcal{N}$$

You should match the form (and amount of randomness) of $\mathcal{N}$ to the problem. It is not critical to the theory what the precise form of $\mathcal{N}$ is, just that it converts the fixed action choice given by $\mu$ into an exploring choice. However, the degree and pattern of exploration will affect speed of learning.

In the paper Continuous Control with Deep Reinforcement Learning, the function $\mathcal{N}$ was constructed using a correlated noise function called Ornstein-Uhlenbeck process, and the paper claims that the auto-correlation this caused between timesteps was a key driver towards faster learning. In practice this choice for $\mathcal{N}$ means that the behaviour policy "drifts" around the current target policy more smoothly than if a simple random sample offset had been used.

• Thank you:) So my misunderstanding is mixing up $\mu$ and $\pi$. The first one is a mapping function, the second is a probability distribution. Commented Apr 22, 2018 at 22:42
• @SherwinChen: Sort of. It is normal to use $\pi$ to denote a policy, but its structure can change depending on the problem. For all intents and purposes, the function $\mu(s)$ is the learned policy if written as $\pi(s)$. However, there is a common notation $\pi(a|s)$ that implies a probability distribution. That is not the case here. I think the authors could well of used $\mu$ in order to avoid any confusion there. Commented Apr 23, 2018 at 7:27
• @tryingtolearn Yeah, I see the difference. Thx :) Commented Aug 6, 2018 at 23:31