2
$\begingroup$

The expected average reward for a policy $\pi$ is: $$ \rho_\pi = \lim_{T \rightarrow \infty } \frac{1}{T} \sum_{t=1}^{T} r_t$$ where $r_t$ is the reward obtained at time $t$ following policy $\pi$.

For a stationary policy $\pi$, $$ \rho_\pi + \lambda_\pi(s) = \bar{r}(s, \pi(s)) + \sum_{s'} p(s'|s,a) \cdot \lambda_\pi(s') $$ where $\bar{r}(s,a)$ and $p(\cdot|s,a)$ are the mean rewards and transition probabilities of the MDP respectively and $\lambda_\pi$ is the bias vector of $\pi$.

Does a similar relation exist for a non-stationary policy?

Edit- Reference for bias vector: The Handbook of Markov Decision Processes 1 defines bias of a stationary deterministic policy as follows:

$\lambda_\pi(s) = \sum_{n = 0}^{\infty} \mathbb{E}[\bar{r}(s_n, \pi(s_n)) - \rho_\pi | s_0 = s]$

where n indicates the current time step and $s_n$ indicates the state at time step $n$ after following policy $\pi$ starting from the initial state $s_0 = s$

$\endgroup$
  • $\begingroup$ Is the non-stationary policy following a specific formula or have any constraints applied to how it can evolve over time? $\endgroup$ – Neil Slater May 20 at 20:50
  • 1
    $\begingroup$ Hi @NeilSlater, no but if having any constraints on the non-stationarity leads to a relation between average reward and single step mean reward, then the constraints could be accommodated. $\endgroup$ – Mathias_Sinner May 21 at 8:15
  • 1
    $\begingroup$ The Handbook of Markov Decision Processes [1] defines bias of a stationary deterministic policy as follows: $\lambda(s, \pi) = \sum_{n = 0}^{\infty} E[\bar{r}(s_n, \pi(s_n)) - \rho_\pi]$ [1]: webee.technion.ac.il/~adam/MDPHandBook/index.html $\endgroup$ – Mathias_Sinner May 21 at 9:06
  • 1
    $\begingroup$ n indicates the current time step. $s_n$ indicates the state at time step $n$ after following policy $\pi$ starting from the initial state $s_0 = s$ $\endgroup$ – Mathias_Sinner May 21 at 9:16
  • $\begingroup$ Thanks for the explanation, it is clearer to me now. $\endgroup$ – Neil Slater May 21 at 9:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.