1
$\begingroup$

The question that follows is from a machine learning textbook (Reinforcement learning Suttion and Barto page 39 link).

Given:

  • a probability distribution over actions $x$ (a policy) at time $t$ denoted $\pi_t(x)$
  • the expected reward, $R_t$, at time $t$ immediately after taking action $a_t$ denoted $q_*(a_t) = \mathbb{E}[R_t | a_t] $.

The following equality is made, $$\mathbb{E}_{A_t}[(q_*(A_t)-B_t)\frac{\partial\pi_t(x)}{\partial H_t(a)}/\pi_t(A_t)]$$ $$=\mathbb{E}_{A_t}[(R_t-B_t)\frac{\partial\pi_t(x)}{\partial H_t(a)}/\pi_t(A_t)]$$

The authors say that the substitution $q_*(A_t) \to R_t $ is valid as $q_*(A_t)=\mathbb{E}[R_t | A_t]$. how does this fact permit the above substitution?

$\endgroup$

2 Answers 2

0
$\begingroup$

Equality comes from two properties of conditional expectation:

  1. Adam's law: $$E_XE[Y|X]=E[Y]$$ 2)"taking out what's known": for any function $h$, $$E[h(X)Y|X] = h(X)E[Y|X]$$

Then $$\mathbb{E}_{A_t}[q_*(A_t)\frac{\partial\pi_t(x)}{\partial H_t(a)}/\pi_t(A_t)]$$ $$=\mathbb{E}_{A_t}[\mathbb{E}[R_t | A_t]\frac{\partial\pi_t(x)}{\partial H_t(a)}/\pi_t(A_t)]$$ $$=\mathbb{E}_{A_t}[\mathbb{E}[R_t \frac{\partial\pi_t(x)}{\partial H_t(a)}/\pi_t(A_t)| A_t]]\hspace{2mm}\text{(by property 2)}$$

$$=\mathbb{E}[R_t \frac{\partial\pi_t(x)}{\partial H_t(a)}/\pi_t(A_t)]\hspace{2mm}\text{(by property 1)}$$

$\endgroup$
0
$\begingroup$

Since you already understood and arrived at $q_*(A_t)=\mathbb{E}[R_t | A_t]$, the next step is to understand $q_*(A_t)$ and $\mathbb{E}[R_t | A_t]$ are actually the same dependent random variable as seemingly differently expressed functions of the same independent random variable $A_t$ here. Finally since you have an outer expectation operator of a function over the distribution of $A_t$, you can simplify $\mathbb{E}[R_t | A_t]$ to the random variable $R_t$ inside the said outer expectation per the famous law of total expectation, aka Adam’s law.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.