Weird results of Q-learning with Softmax

I am implementing an N-armed-bandit with Q-learning. This bandit uses Softmax as its action selection strategy.

This bandit can choose between 4 arms, of which the rewards are distributed as a Normal distribution with the following means and standard deviations:

means = [2.3, 2.1, 1.5, 1.3]
stds =  [0.6, 0.9, 2.0, 0.4]


The bandit plays 1000 games and this is repeated 100 times and averaged.

My code for Softmax is the following:

def play_strategy(self):
tau = self.tau
probabilities = np.zeros(self.N)
for i in range(self.N):
nom = math.exp(self.Qs[self.time_step,i] / tau)
denom = sum(math.exp(val/tau) for val in self.Qs[self.time_step,:])
probabilities[i] = float(nom / denom)
action = np.random.choice(range(self.N), p=probabilities)
return action


For high temperatures ($\tau\to \infty$), all actions have nearly the same probability. For a low temperature ($\tau\to 0^+$), the probability of the action with the highest expected reward tends to 1.
My results seem to show an opposite result. For $\tau=1$, the action with highest expected reward is played most often and for $\tau=0.1$, the actions are played more or less equally.
In addition to @e2crawfo's answer, you could try increasing the number of plays when $\tau$ is small. You have not mentioned how you update the $Q(a,s)$ values, specifically what is the learning rate you are using, perhaps it is too large?