# What is the relation between expected average reward and single step mean reward for a non-stationary MDP policy?

The expected average reward for a policy $$\pi$$ is: $$\rho_\pi = \lim_{T \rightarrow \infty } \frac{1}{T} \sum_{t=1}^{T} r_t$$ where $$r_t$$ is the reward obtained at time $$t$$ following policy $$\pi$$.

For a stationary policy $$\pi$$, $$\rho_\pi + \lambda_\pi(s) = \bar{r}(s, \pi(s)) + \sum_{s'} p(s'|s,a) \cdot \lambda_\pi(s')$$ where $$\bar{r}(s,a)$$ and $$p(\cdot|s,a)$$ are the mean rewards and transition probabilities of the MDP respectively and $$\lambda_\pi$$ is the bias vector of $$\pi$$.

Does a similar relation exist for a non-stationary policy?

Edit- Reference for bias vector: The Handbook of Markov Decision Processes 1 defines bias of a stationary deterministic policy as follows:

$$\lambda_\pi(s) = \sum_{n = 0}^{\infty} \mathbb{E}[\bar{r}(s_n, \pi(s_n)) - \rho_\pi | s_0 = s]$$

where n indicates the current time step and $$s_n$$ indicates the state at time step $$n$$ after following policy $$\pi$$ starting from the initial state $$s_0 = s$$

• Is the non-stationary policy following a specific formula or have any constraints applied to how it can evolve over time? – Neil Slater May 20 at 20:50
• Hi @NeilSlater, no but if having any constraints on the non-stationarity leads to a relation between average reward and single step mean reward, then the constraints could be accommodated. – Mathias_Sinner May 21 at 8:15
• The Handbook of Markov Decision Processes [1] defines bias of a stationary deterministic policy as follows: $\lambda(s, \pi) = \sum_{n = 0}^{\infty} E[\bar{r}(s_n, \pi(s_n)) - \rho_\pi]$ [1]: webee.technion.ac.il/~adam/MDPHandBook/index.html – Mathias_Sinner May 21 at 9:06
• n indicates the current time step. $s_n$ indicates the state at time step $n$ after following policy $\pi$ starting from the initial state $s_0 = s$ – Mathias_Sinner May 21 at 9:16
• Thanks for the explanation, it is clearer to me now. – Neil Slater May 21 at 9:21