I am implementing an N-armed-bandit with Q-learning. This bandit uses Softmax as its action selection strategy.
This bandit can choose between 4 arms, of which the rewards are distributed as a Normal distribution with the following means and standard deviations:
means = [2.3, 2.1, 1.5, 1.3] stds = [0.6, 0.9, 2.0, 0.4]
The bandit plays 1000 games and this is repeated 100 times and averaged.
My code for Softmax is the following:
def play_strategy(self): tau = self.tau probabilities = np.zeros(self.N) for i in range(self.N): nom = math.exp(self.Qs[self.time_step,i] / tau) denom = sum(math.exp(val/tau) for val in self.Qs[self.time_step,:]) probabilities[i] = float(nom / denom) action = np.random.choice(range(self.N), p=probabilities) return action
Now, I've read the following about the Softmax function:
For high temperatures ($\tau\to \infty$), all actions have nearly the same probability. For a low temperature ($\tau\to 0^+$), the probability of the action with the highest expected reward tends to 1.
My results seem to show an opposite result. For $\tau=1$, the action with highest expected reward is played most often and for $\tau=0.1$, the actions are played more or less equally.
What could be the cause of this?