What are soft policies in reinforcement learning? Do soft policies use soft-max function as $\pi(s, a)$ instead of deterministic policies?


what are soft policies in reinforcement learning?

A "soft" policy is one that has some, usually small but finite, probability of selecting any possible action. Having a policy which has some chance of selecting any action is important theoretically when rewards and/or state transitions are stochastic - you are never 100% certain of your estimates for the true value of an action. Soft policies are important for practical purposes of exploring alternative actions, and they can give theoretical guarantees of convergence for RL algorithms.

Does it mean using soft-max function as π(s,a) instead of deterministic policies?

This is one way to create a soft policy. Another very common approach is $\epsilon$-greedy action selection over $Q(s,a)$, where the action with the highest value estimate is used preferentially with $p=1-\epsilon$, or with $p=\epsilon$ a random action is chosen with equal chance of any action.

You may also see the term $\epsilon$-soft policy, which is a policy where every action has at least $p=\frac{\epsilon}{|\mathcal{A}|}$ chance of being selected. The $\epsilon$-greedy policy is also an $\epsilon$-soft policy, but a softmax function will not be in general (depending on what features you are using as input to the softmax).

| cite | improve this answer | |
  • 1
    $\begingroup$ Thanks, so, are soft policies stochastic policies? Since at each state, the policy doesn't have the probability of 1 to choose a deterministic action since chances exist when other actions are chosen. $\endgroup$ – GoingMyWay Apr 24 '18 at 9:22
  • 1
    $\begingroup$ @AlexanderYau Yes soft policies are always stochastic. $\endgroup$ – Neil Slater Apr 24 '18 at 9:40
  • $\begingroup$ Sorry, I mean are soft policies also called stochastic policies? $\endgroup$ – GoingMyWay Apr 24 '18 at 9:46
  • 3
    $\begingroup$ @AlexanderYau: Yes, but not all stochastic policies are soft policies. For instance if you have three actions $\{a,b,c\}$ then a policy $\pi(a)=0.5, \pi(b)=0.5, \pi(c)=0.0$ is a stochastic policy, but it is not a soft policy. So the terms soft policy and stochastic policy are not interchangeable $\endgroup$ – Neil Slater Apr 24 '18 at 9:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.