What's done with the expectations in this proof?

Question

This is a proof of the per-decision importance sampling (theorem 1) from the appendix of:

https://www.google.co.uk/url?sa=t&source=web&rct=j&url=http://scholarworks.umass.edu/cgi/viewcontent.cgi%3Farticle%3D1079%26context%3Dcs_faculty_pubs&ved=2ahUKEwjixPHjgafdAhXIzaQKHah5Ai0QFjAAegQIAxAB&usg=AOvVaw3sN_Eh-1yHqGBtfSoFY0Sx

I should give some background that this is deriving a Monte Carlo importance sampling theorem in reinforcement learning.

I am stuck at the step in brackets where the expectation is split into a product. I know this is possible for independent variables inside the expectation. Is it just that policies after the reward are independent of the reward and the policies are all independent by the Markov assumption and conditioning on state??? If so some insight as to this would be much appreciated because I don't understand how the earlier policies are not independent.

Or maybe it's something else.

Any insight much appreciated!

Many thanks.

Answer 1

Is it just that policies after the reward are independent of the reward and the policies are all independent by the Markov assumption and conditioning on state???

Yes, this is indeed the case.

In this paper, note that the $\pi_i$ and $b_i$ terms do not refer to different policies. As described right above Equation (3) in the paper, they are used as shorthand notation for $\pi(s_t, a_t)$ and $b(s_t, a_t)$, respectively, where $\pi$ is the target policy (the policy about which we wish to learn), and $b$ is the behaviour policy.

These equations are about the case where, using the behaviour policy $b$, we generate a complete trajectory of $T$ time steps (a complete episode). So, across the entire episode, the behaviour policy $b$ and the target policy $\pi$ are consistently the same. These policies are completely fixed and determined before we start the episode, so they cannot be dependent on any of the rewards observed during that episode.

For both of the policies, $\pi(s_t, a_t)$ and $b(s_t, a_t)$ denote the probability assigned to the action that we happened to select in practice ($a_t$) when presented with a state $s_t$. Such probabilities are typically only dependent on precisely the two things you see in parentheses; $s_t$ and $a_t$. We normally do not have policies that are also dependent on earlier history of states, actions or policies (this is indeed not necessary due to the assumption of the Markov property).