Why not sample action from Q values?

Question

When collecting experience from which to estimate a Q(s,a) function, one common technique in the literature is to follow an epsilon greedy-strategy. In this strategy, the agent selects a random action with a probability of epsilon, and the action associated with the maximum Q value otherwise.

Alternatively, why not treat the Q values as a distribution from which to sample an explorative action? Eg: Take the softmax of the q values, and then sample from the softmax distribution to select the action.

Since Q-Learning is off-policy, it should be mathematically sound to do so. Are there any studies where this was considered?

Answer 1

Yes, it's a common thing to do so it's called Boltzmann exploration. It's like softmax but you have additional temperature hyperparameter $T$. The full distribution is \begin{equation} p(a) = \frac{e^{\frac{Q(s,a)}{T}}}{\sum_{a'} e^{\frac{Q(s,a')}{T}}} \end{equation} Hyperparameter $T$ decides how random/greedy you want to be. For large values of $T$ distribution will tend to uniform distribution which means we are randomly picking actions no matter how large their Q-values are. For small value $T$ we will tend to be greedy and pick action with highest Q-value