8
$\begingroup$

What are soft policies in reinforcement learning? Do soft policies use soft-max function as $\pi(s, a)$ instead of deterministic policies?

$\endgroup$

1 Answer 1

9
$\begingroup$

what are soft policies in reinforcement learning?

A "soft" policy is one that has some, usually small but finite, probability of selecting any possible action. Having a policy which has some chance of selecting any action is important theoretically when rewards and/or state transitions are stochastic - you are never 100% certain of your estimates for the true value of an action. Soft policies are important for practical purposes of exploring alternative actions, and they can give theoretical guarantees of convergence for RL algorithms.

Does it mean using soft-max function as π(s,a) instead of deterministic policies?

This is one way to create a soft policy. Another very common approach is $\epsilon$-greedy action selection over $Q(s,a)$, where the action with the highest value estimate is used preferentially with $p=1-\epsilon$, or with $p=\epsilon$ a random action is chosen with equal chance of any action.

You may also see the term $\epsilon$-soft policy, which is a policy where every action has at least $p=\frac{\epsilon}{|\mathcal{A}|}$ chance of being selected. The $\epsilon$-greedy policy is also an $\epsilon$-soft policy, but a softmax function will not be in general (depending on what features you are using as input to the softmax).

$\endgroup$
4
  • 1
    $\begingroup$ Thanks, so, are soft policies stochastic policies? Since at each state, the policy doesn't have the probability of 1 to choose a deterministic action since chances exist when other actions are chosen. $\endgroup$
    – GoingMyWay
    Apr 24, 2018 at 9:22
  • 1
    $\begingroup$ @AlexanderYau Yes soft policies are always stochastic. $\endgroup$ Apr 24, 2018 at 9:40
  • $\begingroup$ Sorry, I mean are soft policies also called stochastic policies? $\endgroup$
    – GoingMyWay
    Apr 24, 2018 at 9:46
  • 3
    $\begingroup$ @AlexanderYau: Yes, but not all stochastic policies are soft policies. For instance if you have three actions $\{a,b,c\}$ then a policy $\pi(a)=0.5, \pi(b)=0.5, \pi(c)=0.0$ is a stochastic policy, but it is not a soft policy. So the terms soft policy and stochastic policy are not interchangeable $\endgroup$ Apr 24, 2018 at 9:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.