1
$\begingroup$

in Sutton's book Reinforcement Learning: An Introduction Chapter 9, how to drive the formula for "The on-policy distribution in episodic tasks" as flow? enter image description here

that h(s) denotes the probability that an episode begins in each state s, and let η(s) denote the number of time steps spent, on average, in state s in a single episode. Time is spent in a state s if episodes start in s, or if transitions are made into s from a preceding state ¯s in which time is spent:

$\endgroup$
1
$\begingroup$

it sort of follows from the definitions of an MDP. The equation simply says that the number of visits to a state $s$ is the sum of the probability of starting in state $s$ and the probability of moving to state $s$ from state $\hat{s}$.

$p(s|\hat{s},a)$ denotes the probability of moving into state $s$ conditional on being in state $\hat{s}$ and performing action $a$. It is essentially a model of the environment where the next state ($s$) can be predicted from the current state ($\hat{s}$) and the currently performed action which is determined by the policy ($\pi(a|\hat{s})$), which determines from the the probability of performing action $a$ given in state $\hat{s}$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.