Weird results of Q-learning with Softmax

Question

I am implementing an N-armed-bandit with Q-learning. This bandit uses Softmax as its action selection strategy.

This bandit can choose between 4 arms, of which the rewards are distributed as a Normal distribution with the following means and standard deviations:

means = [2.3, 2.1, 1.5, 1.3]
stds =  [0.6, 0.9, 2.0, 0.4]

The bandit plays 1000 games and this is repeated 100 times and averaged.

My code for Softmax is the following:

def play_strategy(self):
    tau = self.tau
    probabilities = np.zeros(self.N)
    for i in range(self.N):
        nom = math.exp(self.Qs[self.time_step,i] / tau)
        denom = sum(math.exp(val/tau) for val in self.Qs[self.time_step,:])
        probabilities[i] = float(nom / denom)
    action = np.random.choice(range(self.N), p=probabilities)
    return action

Now, I've read the following about the Softmax function:

For high temperatures ($\tau\to \infty$), all actions have nearly the same probability. For a low temperature ($\tau\to 0^+$), the probability of the action with the highest expected reward tends to 1.

My results seem to show an opposite result. For $\tau=1$, the action with highest expected reward is played most often and for $\tau=0.1$, the actions are played more or less equally.

What could be the cause of this?

Answer 1

This is quite late, but I'll take a stab anyway for posterity. This is likely caused by insufficient exploration when the temperature is low. Because the temperature is low, the algorithm has a high probability of picking the action that it currently thinks is the best. Consequently, if it starts off with a poor initial guess for the Q-values, it is going to take much longer to correct those Q-values because it explores (i.e. tries actions that are sub-optimal according to current estimates for Q-values) at a lower rate, and therefore converges on the correct Q-values much more slowly. The typical solution is to start with a high temperature to explore widely, and reduce it over time to exploit (the opposite of explore) once we've learned reasonably good Q-values.

Answer 2

In addition to @e2crawfo's answer, you could try increasing the number of plays when $\tau$ is small. You have not mentioned how you update the $Q(a,s)$ values, specifically what is the learning rate you are using, perhaps it is too large?