# Why not sample action from Q values?

When collecting experience from which to estimate a Q(s,a) function, one common technique in the literature is to follow an epsilon greedy-strategy. In this strategy, the agent selects a random action with a probability of epsilon, and the action associated with the maximum Q value otherwise.

Alternatively, why not treat the Q values as a distribution from which to sample an explorative action? Eg: Take the softmax of the q values, and then sample from the softmax distribution to select the action.

Since Q-Learning is off-policy, it should be mathematically sound to do so. Are there any studies where this was considered?

Yes, it's a common thing to do so it's called Boltzmann exploration. It's like softmax but you have additional temperature hyperparameter $$T$$. The full distribution is $$$$p(a) = \frac{e^{\frac{Q(s,a)}{T}}}{\sum_{a'} e^{\frac{Q(s,a')}{T}}}$$$$ Hyperparameter $$T$$ decides how random/greedy you want to be. For large values of $$T$$ distribution will tend to uniform distribution which means we are randomly picking actions no matter how large their Q-values are. For small value $$T$$ we will tend to be greedy and pick action with highest Q-value