9
$\begingroup$

In reinforcement learning, is a policy always deterministic, or is it a probability distribution over actions (from which we sample)? If the policy is deterministic, why is not the value function, which is defined at a given state for a given policy $\pi$ as follows

$$V^{\pi}(s) = E\left[\sum_{t>0} \gamma^{t}r_t|s_0 = s, \pi\right]$$

a point output?

In the above definition, we take an expectation. What is this expectation over?

Can a policy lead to different routes?

Improve this question
$\endgroup$
1

2 Answers 2

Reset to default
8
$\begingroup$

There are multiple questions here: 1. Is a policy always deterministic? 2. If the policy is deterministic then shouldn't the value also be deterministic? 3. What is the expectation over in the value function estimate? Your last question is not very clear "Can a policy lead to routes that have different current values?" but I think you mean: 4. Can a policy lead to different routes?

  1. A policy is a function can be either deterministic or stochastic. It dictates what action to take given a particular state. The distribution $\pi(a\mid s)$ is used for a stochastic policy and a mapping function $\pi:S \rightarrow A$ is used for a deterministic policy, where $S$ is the set of possible states and $A$ is the set of possible actions.

  2. The value function is not deterministic. The value (of a state) is the expected reward if you start at that state and continue to follow a policy. Even if the policy is deterministic the reward function and the environment might not be.

  3. The expectation in that formula is over all the possible routes starting from state $s$. Usually, the routes or paths are decomposed into multiple steps, which are used to train value estimators. These steps can be represented by the tuple $(s,a,r,s')$ (state, action, reward, next state)

  4. This is related to answer 2, the policy can lead to different paths (even a deterministic policy) because the environment is usually not deterministic.

Improve this answer
$\endgroup$
8
  • $\begingroup$ can you give me an example of environment not being deterministic? as i see it, if the agent applies action $a$ to an environment in state $s$, it deterministically changes environment to $s^`$ $\endgroup$
    – figs_and_nuts
    Commented Dec 1, 2017 at 20:53
  • 1
    $\begingroup$ A classical example is a robot that takes moves left by one step (action) but the surface is slippery (walking on ice) so it actually moves 2 steps left. In fact such environments are the norm and extensively studied. My example is actually as well known "toy" environment :gym.openai.com/envs/FrozenLake-v0 $\endgroup$
    – A.D
    Commented Dec 1, 2017 at 20:59
  • $\begingroup$ so state $s$ and action $a$ upon it leads to a prob dist over $s^`$. i got that right? $\endgroup$
    – figs_and_nuts
    Commented Dec 1, 2017 at 21:04
  • $\begingroup$ Yes, just like $p(a\mid s)$ is stochastic, $p(s' \mid s, a)$ is also stochastic. $\endgroup$
    – A.D
    Commented Dec 1, 2017 at 21:05
  • $\begingroup$ just two more things: 1. $p(a|s)$ is stochastic only in stochastic policy right? 2. Can you confirm the other answer posted is wrong about what the expectation is taken over part so i can accept your answer? $\endgroup$
    – figs_and_nuts
    Commented Dec 1, 2017 at 21:11
1
$\begingroup$

The policy can be stochastic or deterministic. The expectation is over training examples given the conditions. The value function is an estimate of the return, which is why it's an expectation.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.