In the Q-learning method to get the optimal strategy, the update method is like the following: \begin{equation} Q(S,A) \leftarrow \ Q(S,A) + \alpha [R+\gamma~max_a(Q(s',a)) -Q(S,A)] \end{equation}

If in Q-learning the state transition probabilities are fixed to 1 then during the policy exploration how do we consider the $\epsilon$-greedy algorithm? As $\epsilon$ denotes the small probability during exploration and it must not be 1. Are the state transition probabilities and the probability during policy exploration different?


I would recommend reading Chapter 3 of Sutton and Barto's RL book. In it, you will learn about how reinforcement learning problems are formulated as a series of interactions between an agent and its environment. This is the key diagram.

enter image description here

The "agent" is the entity that makes decisions. The "environment" is everything else, defined in the book as "everything outside of the agent's absolute control."

Notice that the environment responds to the agent's action with a reward $R_{t+1}$ and a next state $S_{t+1}$. Both of these are, in general, random variables with probability distributions conditioned on action $A_t$ and current state $S_t$.

In its attempt to behave optimally in a stochastic or uncertain environment, an agent has the option to introduce randomness of its own in the form of a stochastic policy. Here, a policy is a distribution over possible actions conditioned on the agent's current state. The agent samples from this distribution to decide what action to take.

Q-learning is a popular algorithm in RL, but I would suggest you start with reading the fundamentals (in the book I mentioned) before trying to implement any RL algorithms.

  • $\begingroup$ Thank you for your comments. I am going through the given book. $\endgroup$
    – A Paul
    Mar 8 '19 at 6:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.