I am implementing an N-armed-bandit with Q-learning. This bandit uses Softmax as its action selection strategy.

This bandit can choose between 4 arms, of which the rewards are distributed as a Normal distribution with the following means and standard deviations:

means = [2.3, 2.1, 1.5, 1.3]
stds =  [0.6, 0.9, 2.0, 0.4]

The bandit plays 1000 games and this is repeated 100 times and averaged.

My code for Softmax is the following:

def play_strategy(self):
    tau = self.tau
    probabilities = np.zeros(self.N)
    for i in range(self.N):
        nom = math.exp(self.Qs[self.time_step,i] / tau)
        denom = sum(math.exp(val/tau) for val in self.Qs[self.time_step,:])
        probabilities[i] = float(nom / denom)
    action = np.random.choice(range(self.N), p=probabilities)
    return action

Now, I've read the following about the Softmax function:

For high temperatures ($\tau\to \infty$), all actions have nearly the same probability. For a low temperature ($\tau\to 0^+$), the probability of the action with the highest expected reward tends to 1.

My results seem to show an opposite result. For $\tau=1$, the action with highest expected reward is played most often and for $\tau=0.1$, the actions are played more or less equally.

What could be the cause of this?