Why do we use expectation in reinforcement learning?

In reinforcement learning we use lot of expectations, what's the logic behind them?

Like in reward function, we use expected return instead of return, in value functions V and action-value functions Q we use expectation. Almost every-term in reinforcement learning is written in the form of expectation. Why?

Why do we use expectations?

• Also posted on CS.SE. Please do not post the same question on multiple sites. Each community should have an honest shot at answering without anybody's time being wasted. – D.W. Jul 23 '16 at 15:03
• @atayenel, in the future, if you're going to suggest another site, it'd be nice if you could remind people not to cross-post. Otherwise the natural inclination is to re-post another copy (which is against SE rules) and that just leads to a bad experience all around. Thank you -- and thanks for trying to help Gary! – D.W. Jul 23 '16 at 15:03

Because life is uncertain. We don't know what the future might hold.

If we knew the future, we'd calculate the reward we'll receive for each possible action, and choose the best one -- but alas, no one can tell the future. Therefore, we can't be sure what the reward of each possible action will be, and we can't be sure which action is best.

So, instead, for each action, we calculate the average of all possible rewards (weighted by their likelihood). Roughly speaking, this is our best guess at what the reward of the future might be, given the information available to us right now and the unavoidable uncertainty about the future. Then, we use that information to guide our decisions.

You can view this as a consequence of how we define "optimal" in most reinforcement learning applications: An optimal policy is that which maximizes expected discounted reward in a Markov decision process. MDPs are RL's core and longest-studied problem, making them a natural starting point.

Though natural, this definition may not fit every application. Generalized MDPs replace the $\max\limits_a$ and $\mathop{\mathbb{E}}\limits_{s'}$ operators with other non-expansions. For example, replace $\mathop{\mathbb{E}}\limits_{s'}$ with $\min\limits_{s'}$, and you have a risk-sensitive MDP.

Several standard planning and learning algorithms—value iteration, policy iteration, model-based RL and Q learning—can be generalized to work in this framework. (Szepesvári and Littman.)

We use expectations because we want to optimize the long-term performance of our algorithms. This is the weighted sum of all possible outcomes multiplied by their probabilities — the expected reward.