# how Deriving the formula for “The on-policy distribution in episodic tasks”?

in Sutton's book Reinforcement Learning: An Introduction Chapter 9, how to drive the formula for "The on-policy distribution in episodic tasks" as flow? that h(s) denotes the probability that an episode begins in each state s, and let η(s) denote the number of time steps spent, on average, in state s in a single episode. Time is spent in a state s if episodes start in s, or if transitions are made into s from a preceding state ¯s in which time is spent:

## 1 Answer

it sort of follows from the definitions of an MDP. The equation simply says that the number of visits to a state $$s$$ is the sum of the probability of starting in state $$s$$ and the probability of moving to state $$s$$ from state $$\hat{s}$$.

$$p(s|\hat{s},a)$$ denotes the probability of moving into state $$s$$ conditional on being in state $$\hat{s}$$ and performing action $$a$$. It is essentially a model of the environment where the next state ($$s$$) can be predicted from the current state ($$\hat{s}$$) and the currently performed action which is determined by the policy ($$\pi(a|\hat{s})$$), which determines from the the probability of performing action $$a$$ given in state $$\hat{s}$$.