# In this RL problem, why is the substitution $q_*(A_t)=\mathbb{E}[R_t | A_t] \to R_t$ valid within this expectation (over actions)?

The question that follows is from a machine learning textbook (Reinforcement learning Suttion and Barto page 39 link).

Given:

• a probability distribution over actions $$x$$ (a policy) at time $$t$$ denoted $$\pi_t(x)$$
• the expected reward, $$R_t$$, at time $$t$$ immediately after taking action $$a_t$$ denoted $$q_*(a_t) = \mathbb{E}[R_t | a_t]$$.

The following equality is made, $$\mathbb{E}_{A_t}[(q_*(A_t)-B_t)\frac{\partial\pi_t(x)}{\partial H_t(a)}/\pi_t(A_t)]$$ $$=\mathbb{E}_{A_t}[(R_t-B_t)\frac{\partial\pi_t(x)}{\partial H_t(a)}/\pi_t(A_t)]$$

The authors say that the substitution $$q_*(A_t) \to R_t$$ is valid as $$q_*(A_t)=\mathbb{E}[R_t | A_t]$$. how does this fact permit the above substitution?

## 2 Answers

Equality comes from two properties of conditional expectation:

1. Adam's law: $$E_XE[Y|X]=E[Y]$$ 2)"taking out what's known": for any function $$h$$, $$E[h(X)Y|X] = h(X)E[Y|X]$$

Then $$\mathbb{E}_{A_t}[q_*(A_t)\frac{\partial\pi_t(x)}{\partial H_t(a)}/\pi_t(A_t)]$$ $$=\mathbb{E}_{A_t}[\mathbb{E}[R_t | A_t]\frac{\partial\pi_t(x)}{\partial H_t(a)}/\pi_t(A_t)]$$ $$=\mathbb{E}_{A_t}[\mathbb{E}[R_t \frac{\partial\pi_t(x)}{\partial H_t(a)}/\pi_t(A_t)| A_t]]\hspace{2mm}\text{(by property 2)}$$

$$=\mathbb{E}[R_t \frac{\partial\pi_t(x)}{\partial H_t(a)}/\pi_t(A_t)]\hspace{2mm}\text{(by property 1)}$$

Since you already understood and arrived at $$q_*(A_t)=\mathbb{E}[R_t | A_t]$$, the next step is to understand $$q_*(A_t)$$ and $$\mathbb{E}[R_t | A_t]$$ are actually the same dependent random variable as seemingly differently expressed functions of the same independent random variable $$A_t$$ here. Finally since you have an outer expectation operator of a function over the distribution of $$A_t$$, you can simplify $$\mathbb{E}[R_t | A_t]$$ to the random variable $$R_t$$ inside the said outer expectation per the famous law of total expectation, aka Adam’s law.