0
$\begingroup$

In the second edition of the book "Reinforcement Learning: an introduction" by Sutton and Bato page 324 (Policy gradient chapter): It says that: Given a state, the effect of the policy parameter on the actions, and thus on reward, can be computed in a relatively straightforward way from knowledge of the parameterization. But the effect of the policy on the state distribution is a function of the environment and is typically unknown.

Can anyone explain why it is straightforward for actions and why it is unknown for state distribution? Thank you.

$\endgroup$

2 Answers 2

1
$\begingroup$

@AliGh What the author means by "the effect of the policy on the state distribution is a function of the environment and is typically unknown" is that to compute the state distribution besides the policy you also require transition model of the environment and this transition model is usually unknown.

Chelsea Finn derives a dynamic programming algorithm to compute state distribution on blackboard in lecture 10b of deep rl bootcamp. You can look at it for further clarification.

Lecture 10b of deep rl bootcamp https://www.youtube.com/watch?v=d9DlQSJQAoI&feature=youtu.be

$\endgroup$
0
$\begingroup$

If you have a neural network policy $\pi_\theta(\cdot | s)$, you can compute the effect of $\theta$ on $\pi$ simply by doing a single forward pass of the network. On the other hand, there is no easy way to compute the distribution of states given that your agent is following the policy $\pi_\theta$. Just try to think of how you would compute it -- there's no good way.

$\endgroup$
5
  • $\begingroup$ But my problem is that what is the point of the author when it says the effect of policy on the state distribution is unknown? Because state distribution is about environment and policy is what agent does, so affecting the environment by the agent cannot happen. $\endgroup$
    – Ali Ghghgh
    Commented Dec 18, 2018 at 6:55
  • $\begingroup$ "Because state distribution is about environment". No, the state distribution is a function of both the environment and the agent $\endgroup$
    – shimao
    Commented Dec 18, 2018 at 6:58
  • $\begingroup$ Based on what you say, the agent should have the power to change the environment. Can you give me an example or reference for more details? $\endgroup$
    – Ali Ghghgh
    Commented Dec 18, 2018 at 7:13
  • $\begingroup$ Consider an environment with three states, left, middle, and right, and each episode starts with the agent in the middle state. The agent has two actions, L and R, which move the agent to the left / right state respectively. The state distribution for an policy which always takes the L action is obviously different from the state distribution of a policy which always takes the R action. $\endgroup$
    – shimao
    Commented Dec 18, 2018 at 7:19
  • $\begingroup$ @AliGh if you mean "the state" by environment, then yes. If you mean the underlying MDP, then no, an agent cannot change that. $\endgroup$
    – shimao
    Commented Dec 18, 2018 at 7:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.