When collecting experience from which to estimate a Q(s,a) function, one common technique in the literature is to follow an epsilon greedy-strategy. In this strategy, the agent selects a random action with a probability of epsilon, and the action associated with the maximum Q value otherwise.
Alternatively, why not treat the Q values as a distribution from which to sample an explorative action? Eg: Take the softmax of the q values, and then sample from the softmax distribution to select the action.
Since Q-Learning is off-policy, it should be mathematically sound to do so. Are there any studies where this was considered?