# Can someone explain how expectations are used and evaluated in reinforcement learning?

Going through some tutorials on reinforcement learning, I cannot help but to notice that people selectively put expectation operators around their equations (value, Q, loss, etc.), and when expectation is used, it is never evaluated.

Note: By "evaluated", I mean, if I were asked in a probability exam to find the expectation of $X$, where $X \sim \mathcal{N}(\mu, \sigma^2)$ is the Gaussian distribution, I would write $E(f_X(x)) = \mu$. If it were the uniform random variable, I would write $E(f_X(x)) = \dfrac{1}{n}$. It would be unthinkable for me to leave it as $E(f_X(x))$, but this is exactly what people in reinforcement learning does.

The usage of the expectation is also irregular:

For example, in these papers, expectation is not used in the definition of state-action value/function.

https://hmjianggatech.github.io/files/HCK.pdf

In these papers, expectations are used

https://arxiv.org/pdf/1511.06581.pdf

http://paulorauber.com/notes/reinforcement_learning.pdf

https://www.nature.com/articles/nature14236.pdf

Finally, it is not clear to me how people can actually implement these expressions as actual algorithms, because the distributions, i.e., the policy, the environment, the reward, are never specified, they are just assumed to exist. More to the point, say that I am given the following definition,

$Q_\pi(s, a) = E_\pi[R_t|s_t = s, a_t = a]$

However, we do not know what $\pi$ is at the beginning, we only know that it is some probability distribution. Furthermore, it is not clear how the inner expressions i.e. $R_t$.... depends on $\pi$. How do we evaluate this expression as something that is actually implementable in code?

In Dynamic Programming, where you have a model, it is possible to expand from an expectation, using known probabilities from the model. Your reference text does this on page 4 for $Q^{\pi}(s,a)$. If you know the policy and environment model, the precise values of Q can be calculated. This is how Policy Iteration and Value Iteration work.
As an aside, often $\pi(a|s)$ is known or under control of the learning agent, so it can be used to partially resolve expectations even when sampling. This knowledge is used in extensions to basic algorithms such as importance sampling and Expected SARSA.