Confusions about how to choose an action in deterministic policy gradient I read "Continuous Control with Deep Reinforcement Learning". it says

The DPG algorithm maintains a parameterized actor function μ(s|θμ) which specifies the current policy by deterministically mapping states to a specific action. 

I thought DPG simply chose the action with maximum $\mu(s|\theta_\mu)$(with noise for exploration). But when I read "Deterministic Policy Gradient Algorithms", it seems to say a different story:

To ensure that our deterministic policy gradient algorithms continue to explore satisfactorily, we introduce an off-policy learning algorithm. The basic idea is to choose actions according to a stochastic behavior policy (to ensure adequate exploration), but to learn about a deterministic target policy (exploiting the efficiency of the deterministic policy gradient)

According to my understanding, it says to choose an action as the stochastic policy gradient, but to learn the policy parameters using $\nabla_\theta Q(s,a)$
I'm not sure whether I've misunderstood something, but now I get confused -- how does DPG select an action exactly? or both above cases work fine?
Thanks in advance.
 A: The function $\mu(s,\theta_{\mu})$ is not a value function, and you cannot take a meaningful maximum over it when deciding which action to take.
In fact the function must return a single action, that's what the Deterministic part of Deterministic Policy Gradient means. In addition the function $\mu(s,\theta_{\mu})$ must be differentiable with respect to $\theta$. So DPG is most easily applied in continuous action problems, where the action is a vector of real values, such as amount of steering and acceleration for a robotic vehicle.

How does DPG select an action exactly?

In order to explore, DPG selects an action by adding an exploring function (noted as $\mathcal{N}$) to the deterministic policy, creating $\mu'(s)$ as the behaviour policy:
$$\mu'(s_t) = \mu(s_t,\theta^{\mu}_t) + \mathcal{N}$$
You should match the form (and amount of randomness) of $\mathcal{N}$ to the problem. It is not critical to the theory what the precise form of $\mathcal{N}$ is, just that it converts the fixed action choice given by $\mu$ into an exploring choice. However, the degree and pattern of exploration will affect speed of learning.
In the paper Continuous Control with Deep Reinforcement Learning, the function $\mathcal{N}$ was constructed using a correlated noise function called Ornstein-Uhlenbeck process, and the paper claims that the auto-correlation this caused between timesteps was a key driver towards faster learning. In practice this choice for $\mathcal{N}$ means that the behaviour policy "drifts" around the current target policy more smoothly than if a simple random sample offset had been used.
