Natural way to construct stochastic policy from value function?

Question

I've been thinking about this question for a while now and I can't seem to convince myself of the (in)validity of deriving a probabilistic policy from a state-action value function $Q(s,a)$.

Of course, one way to derive a stochastic policy from $Q(s,a)$ is to take the epsilon-greedy policy. I would like to be able to do something a bit more sophisticated, though.

why?

One of the reasons I'd like to be able to do this, is that I'd like to do Thompson sampling to pick my next action. Another reason is that I'd like to implement the expected-SARSA algorithm for the stochastic policy derived from $Q(s,a)$ itself.

softmax?

At first, I thought I'd define my policy as the softmax of $Q(s,a)$, i.e.

$$\pi_\text{proba}(a|s)\ \mathop{=}^.\ \text{softmax}_a\, Q(s,a) $$

This was in part inspired by the greedy policy, which uses argmax instead of softmax:

$$\pi_\text{greedy}(a|s)\ \mathop{=}^.\ \mathbb{I}\bigl(a=\text{argmax}_a\, Q(s,a)\bigr) $$

But then I started worrying about the fact that the value function is not meant to represent probabilities (it measures the expected return). At first, I waved it away thinking this is just my definition of $\pi_\text{proba}(a|s)$. I'm free to define the distribution I draw my actions from, aren't I?

z-scaling?

Even though it is true that I can define whatever I like, I would also like my policy to approach an optimum. An objection to using $Q$-values is that they aren't normalized. So I figured I'd just $z$-scale the $Q$-values before feeding them to the softmax function, e.g.

$$z(s,a)\ =\ \bigl(Q(s,a) - \mu\bigr) / \sigma$$

where $\mu=\frac1n\sum_aQ(s,a)$ and $\sigma^2=\frac1{n-1}\sum_a\bigl(Q(s,a) - \mu\bigr)^2$.

First of all, it's easy to show that subtracting off the mean $\mu$ has no effect on the softmax, but rescaling by $\sigma$ makes a big difference. For example, let our actions be binary $a\in\{0,1\}$ and suppose we have $Q(s,0)=Q(s,1)+\varepsilon$ for some $\varepsilon$ arbitrarily small. The resulting normalized $Q$-value will always be $z(s,1)=-z(s,0)=\text{sign}(\varepsilon)/\sqrt{2}$. In other words, there is a discontinuity at $\varepsilon=0$. This is definitely not desired.

suggestions?

At this point I'm stuck. My questions:

1. Does it even make sense to construct a stochastic policy from a value function?
2. If it does, what would be a natural mapping from $Q(s,a)$ to $\pi(a|s)$?

EDIT

FWIW, I implemented Expected-SARSA for gym's CartPole-v0 environment with a straight-up softmax (no z-scaling) and it seems to work (see notebook). Performance isn't impressive, but it works. When I replace epsilon-greedy with Thompson sampling, though, performance drops severely (to the point that it doesn't converge).

Answer 1

For people who stumble upon this question later on, I did find an answer to this question.

The soft policy (or stochastic/probabilistic policy) constructed from a state-action value function in the way I described in my question is called a Boltzmann policy. It is defined as follows: $$ \pi_\text{Boltzmann}(a|s)\ =\ \frac{\text{e}^{Q(s, a) / \tau}}{\sum_b\text{e}^{Q(s, b)/\tau}}\ =\ \text{softmax}_a\left( \frac{Q(s,a)}{\tau} \right) $$ The generalized temperature $\tau$ is a free parameter that needs to be specified for each different application.

To answer my own questions:

1. Is sort of makes sense, yes. See also this blog post.
2. Apart from having to specify $\tau$, the softmax seems to be the natural mapping.