What are soft policies in reinforcement learning? What are soft policies in reinforcement learning? Do soft policies use soft-max function as $\pi(s, a)$ instead of deterministic policies?
 A: 
what are soft policies in reinforcement learning?

A "soft" policy is one that has some, usually small but finite, probability of selecting any possible action. Having a policy which has some chance of selecting any action is important theoretically when rewards and/or state transitions are stochastic - you are never 100% certain of your estimates for the true value of an action. Soft policies are important for practical purposes of exploring alternative actions, and they can give theoretical guarantees of convergence for RL algorithms.

Does it mean using soft-max function as π(s,a) instead of deterministic policies?

This is one way to create a soft policy. Another very common approach is $\epsilon$-greedy action selection over $Q(s,a)$, where the action with the highest value estimate is used preferentially with $p=1-\epsilon$, or  with $p=\epsilon$ a random action is chosen with equal chance of any action.
You may also see the term $\epsilon$-soft policy, which is a policy where every action has at least $p=\frac{\epsilon}{|\mathcal{A}|}$ chance of being selected. The $\epsilon$-greedy policy is also an $\epsilon$-soft policy, but a softmax function will not be in general (depending on what features you are using as input to the softmax).
