Policy Gradient Methods advantages over value-based methods

Question

In the RL bible by Sutton and Barto it says on page 322 regarding the advantages of policy gradient methods:

If the action space is discrete and not too large, then a natural and common kind of parameterization is to form parameterized numerical preferences $h(s, a, \theta) \in \mathbb{R}$ for each state–action pair. The actions with the highest preferences in each state are given the highest probabilities of being selected, for example, according to an exponential softmax distribution.

and

One advantage of parameterizing policies according to the softmax in action preferences is that the approximate policy can approach a deterministic policy, whereas with $\epsilon$-greedy action selection over action values there is always an $\epsilon$ probability of selecting a random action.

My question is now: is this still the case with $\epsilon$-decay? I mean, the deterministic policy in policy gradients is only approached in the limit as the score for the desired action can only reach $\infty$ in the limit (and the scores for all other actions can only reach $\textbf{0}$ in the limit). The same is true for $\epsilon$-greedy with $\epsilon$-decay as $\epsilon$ will reach $\textbf{0}$ in the limit and thereafter the policy will be deterministic.

Any idea if Sutton and Barto are still right regarding $\epsilon$-decay?

Answer 1

I think I just figured it out. The advantage of policy gradients is that approaching a deterministic policy is dependent on the experiences as only certain experiences will push the score of some action towards infinity. With $\epsilon$-greedy however, this is not the case as the decay factor is set externally and is not dependent on the security of the policy or the learned experiences so far. That being said, policy gradients can approach a deterministic policy in the limit if choosing actions deterministically really is the (locally) best option. $\epsilon$-greedy however will certainly approach a deterministic policy as this is preset in advance. However, this deterministic policy might actually not be desirable.