I am implementing an N-armed-bandit with Q-learning. This bandit uses Softmax as its action selection strategy.

This bandit can choose between 4 arms, of which the rewards are distributed as a Normal distribution with the following means and standard deviations:

means = [2.3, 2.1, 1.5, 1.3]
stds =  [0.6, 0.9, 2.0, 0.4]

The bandit plays 1000 games and this is repeated 100 times and averaged.

My code for Softmax is the following:

def play_strategy(self):
    tau = self.tau
    probabilities = np.zeros(self.N)
    for i in range(self.N):
        nom = math.exp(self.Qs[self.time_step,i] / tau)
        denom = sum(math.exp(val/tau) for val in self.Qs[self.time_step,:])
        probabilities[i] = float(nom / denom)
    action = np.random.choice(range(self.N), p=probabilities)
    return action

Now, I've read the following about the Softmax function:

For high temperatures ($\tau\to \infty$), all actions have nearly the same probability. For a low temperature ($\tau\to 0^+$), the probability of the action with the highest expected reward tends to 1.

My results seem to show an opposite result. For $\tau=1$, the action with highest expected reward is played most often and for $\tau=0.1$, the actions are played more or less equally.

enter image description here

What could be the cause of this?


2 Answers 2


This is quite late, but I'll take a stab anyway for posterity. This is likely caused by insufficient exploration when the temperature is low. Because the temperature is low, the algorithm has a high probability of picking the action that it currently thinks is the best. Consequently, if it starts off with a poor initial guess for the Q-values, it is going to take much longer to correct those Q-values because it explores (i.e. tries actions that are sub-optimal according to current estimates for Q-values) at a lower rate, and therefore converges on the correct Q-values much more slowly. The typical solution is to start with a high temperature to explore widely, and reduce it over time to exploit (the opposite of explore) once we've learned reasonably good Q-values.


In addition to @e2crawfo's answer, you could try increasing the number of plays when $\tau$ is small. You have not mentioned how you update the $Q(a,s)$ values, specifically what is the learning rate you are using, perhaps it is too large?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.