In reinforcement learning, we seek an optimal policy which is defined as a mapping $\pi: s \mapsto a$ from the set of states to the set of actions, or a mapping $\pi: s \mapsto p(a|s)$, $p(a|s)$ is a probability distribution over my actions, given a certain state.

However, this almost seems counter-intuitive if not unrealistic for many real-world scenarios. In real-life, it seems that we simultaneously take state and reward into consideration or even just the reward.

Why we don't even need the state

Consider the following, I am playing rock-paper-scissors against an opponent, then when my action is chosen (Rock/Paper/Scissor), if I am beaten, then it is almost as if I receive a negative reward, if I win, a positive reward, if we tie, then no change.

Hence my brain tries to produce a policy $\pi: r \mapsto a$. For instance, if I receive a consecutive negative reward, then I would change my strategy. There is just mapping from reward to action. Defining a state seems to be redundant here.

In a even more extreme case. I don't even know my state. All I have on my screen is a stream of reward (or actual $$), say 1,4,2,5,8,0,3,4. I hit some keys on my keyboard, and the numbers change. I want press key in such a way to get the biggest number. There is no need for a state here. So the mathematical definition of $\pi$ doesn't even make sense because the state space is the empty set!

Why we might need both the state and the reward

From a mathematical perspective, $\pi(s_t) = \text{argmax}_a Q(s_t,a_t)$, but the $Q$ function is defined as a function of the reward! In fact, it is the expectation over the current reward $r_t + \text{ other terms }$

So even from a purely mathematical perspective, $\pi$ should be a function of both the state and the reward, as it simply just depends on $r_t$. That's just composition of functions man!

Am I over thinking this issue? Can someone who is expert in this area let me know how people even think about the policy when 1. the state is not defined, 2. when reward also needs to be taken into account


Tl;dr, I think you're confusing "policy" for "learning algorithm".

Informally speaking, a policy is like your current strategy for playing a game. When you receive some feedback (from either winning or losing the game), then you might adjust and improve your strategy. That is called "learning" a policy or improving a policy. In reinforcement learning, we usually care about learning algorithms which will start with a suboptimal policy, and gradually head towards a good policy.

It's understandable that you're confused by the two examples you gave, because they're kind of degenerate cases. You've basically described a multi-armed bandit problem, which is a special case of the MDP (markov decision process) framework when there is only a single state (the state space is a singleton set, not empty).

Suppose your strategy (policy) is to always choose rock: $\pi(0) = \text{Rock}$, where $0$ is the single state. And your opponent always chooses paper in response, so you decide to act on your negative reward signal and change to a new policy $\pi'(0) = \text{Scissors}$. You could say you have some learning strategy $\pi' \leftarrow L(\pi, \tau)$ which takes the current policy, some trajectories from it, and produces a new and improved policy. However $\pi$ and $\pi'$ are still perfectly valid policies themselves, mapping state to (deterministic) action without relying on reward. Only your learning strategy made use of the reward signal.

$\pi(s_t) = \text{argmax}\ Q(s_t,a_t)$

This is, in general, incorrect. Sure, this is one way to define a policy, but it is not the only way.

but the 𝑄 function is defined as a function of the reward!

To be precise, the $Q$ function is defined with respect to a policy and an environment. So one might write $Q_{\pi,E}$ (although typically the environment is always fixed).

So even from a purely mathematical perspective, 𝜋 should be a function of both the state and the reward, as it simply just depends on 𝑟. That's just composition of functions man!

Consider the equation $x = E_y[x+y]$, where $y$ is a normal random variable with mean $x$. Then it must be that $x = 0$. Somehow, the RHS was defined using $y$, yet we ended up finding that $x$ does not depend on $y$! This is an example of why your reasoning doesn't quite work.

Also to be precise, $Q$ is not defined in terms of $r$ -- it is defined in terms of the policy and the environment. The environment and policy together produce a distribution over rewards / return, not any single reward value.

  • $\begingroup$ Thanks. To your first answer, would you say that most if not all supervised learning are reinforcement learning? Agent in a state produces a prediction (action) and receives a reward (loss value), and the process repeat. $\endgroup$
    – Fraïssé
    Jan 29 '20 at 6:26
  • $\begingroup$ To your second answer, is it correct to think that since the reward is defined in terms of some other variables, e.g., the state, therefore ultimately, the policy is just a function of the state? By environment, you mean the probability transition function associated with the environment, right? $\endgroup$
    – Fraïssé
    Jan 29 '20 at 6:28
  • 1
    $\begingroup$ 1. supervised learning can be interpreted as a reinforcement learning task (although this interpretation isn't very useful). 2. the policy is a function which maps each state to an action or distribution of actions. 3. by environment, I mean some MDP $\endgroup$
    – shimao
    Jan 29 '20 at 6:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.