Multi-armed bandit epsilon greedy

Question

This is the code from a lecture from the Artificial Intelligence Reinforcement Learning in Python course on Udemy to implement the multi-armed bandit epsilon greedy. m is the true mean and mean is the estimated mean.

I don't understand why in the pull function we need to add the new random number to the true mean. We know the real parameters, so there is no need to solve the problem.

import numpy as np
import matplotlib.pyplot as plt


class Bandit:
  def __init__(self, m):
    self.m = m
    self.mean = 0
    self.N = 0

  def pull(self):
    return np.random.randn() + self.m

  def update(self, x):
    self.N += 1
    self.mean = (1 - 1.0/self.N)*self.mean + 1.0/self.N*x


def run_experiment(m1, m2, m3, eps, N):
  bandits = [Bandit(m1), Bandit(m2), Bandit(m3)]

  data = np.empty(N)

  for i in range(N):
    # epsilon greedy
    p = np.random.random()
    if p < eps:
      j = np.random.choice(3)
    else:
      j = np.argmax([b.mean for b in bandits])
    x = bandits[j].pull()
    bandits[j].update(x)

    # for the plot
    data[i] = x
  cumulative_average = np.cumsum(data) / (np.arange(N) + 1)

  # plot moving average ctr
  plt.plot(cumulative_average)
  plt.plot(np.ones(N)*m1)
  plt.plot(np.ones(N)*m2)
  plt.plot(np.ones(N)*m3)
  plt.xscale('log')
  plt.show()

  for b in bandits:
    print(b.mean)

  return cumulative_average

if __name__ == '__main__':
  c_1 = run_experiment(1.0, 2.0, 3.0, 0.1, 100000)
  plt.plot(c_1, label='eps = 0.1')

Answer 1

I don't understand why in "pull" function we need to add the new random number to the true mean?

The result of pulling the lever on the bandit is what the agent observes. So this is simulating the environment, which has a random reward drawn from a distribution.

The agent is deliberately only allowed to call this pull method, and may not for instance query the true mean.

if we know the real parameters so there is no need to solve the problem

Obviously, it is trivial to find the action that maximises expected reward if you know the parameters. It is just the action with the highest mean reward.

The premise of the exercise is that the agent does not know the parameters or - for most basic agents - anything about the distribution of rewards at all.

For these toy problems, you know the parameters because you wrote them as part of the simulation. However, the Bandit class represents an environment which could be something that you don't know distribution of rewards for. For example it might be the outcome of a medical trial using a certain treatment, or the chance of someone clicking through on an advert and spending a certain amount of money.

Most importantly, the code is constructed such that the agent does not know the parameters, and the experiment is to build agents that can find the best action through sampling. In multi armed bandit problems we also want to achieve that efficiently - with a small number of samples, or whilst gaining the most reward we can whilst still learning from the environment (this can also be framed as minimising regret - how the agent compares to a theoretical oracle that does know the parameters from the start).

On a read through the code, the class design is somewhat poor/lazy as a teaching tool, in that the author has stored both the simulation and stats in the same class. It's fine as an implementation, but makes it less clear which parts of the system are the agent and which are the environment. I would personally have used a structure outside of Bandit class to store the current mean estimates that the agent uses.