What is the exploration behavior normally used in Policy Search based Reinforcement Learning? When using Q-learning, we have different options for a behavior policy, for e.g. we can use $\epsilon-$greedy. 
But I have not seen $\epsilon-$greedy prescribed for policy search/gradient methods. When sampling trajectories in "vanilla" policy gradient, it looks like we only use softmax action selection. Is this claim true? If so, why? 
 A: 
When sampling trajectories in "vanilla" policy gradient, it looks like we only use softmax action selection. Is this claim true? If so, why?

It is not strictly true. However, it is a common choice for action selection for discrete actions. The important thing is that your policy function is parametric on some features of the state, and it is possible to take the derivative of the function with respect to the the parameters.
Softmax action selection meets those constraints, and it is has a straightforward gradient with respect to parameters. You cannot use $\epsilon$-greedy, because taking the greedy action is not differentiable. Something like Gibbs/Boltzmann sampling is differentiable, so you could use that - in practice this is so similar to softmax, it is not going to be worth handling the addition of the temperature parameter.
Another common choice, when faced with continuous action space, is to output the parameters of a continuous probability distribution function, such as mean and variance of a Normal distribution.
