# What is predicted and controlled in reinforcement Learning?

In reinforcement learning, I saw many notions with respect to control and prediction, like Monte Carlo prediction and Monte Carlo control.

But what are we actually predicting and controlling?

The difference between prediction and control is to do with goals regarding the policy. The policy describes the way of acting depending on current state, and in the literature is often noted as $\pi(a|s)$, the probability of taking action $a$ when in state $s$.

So, my question is for prediction, predict what?

A prediction task in RL is where the policy is supplied, and the goal is to measure how well it performs. That is, to predict the expected total reward from any given state assuming the function $\pi(a|s)$ is fixed.

for control, control what？

A control task in RL is where the policy is not fixed, and the goal is to find the optimal policy. That is, to find the policy $\pi(a|s)$ that maximises the expected total reward from any given state.

A control algorithm based on value functions (of which Monte Carlo Control is one example) usually works by also solving the prediction problem, i.e. it predicts the values of acting in different ways, and adjusts the policy to choose the best actions at each step. As a result, the output of the value-based algorithms is usually an approximately optimal policy and the expected future rewards for following that policy.

• Just keep in mind "RL" is pretty ill-defined. Usually people mean "model-free full RL" where there is no model for the rewards, no model for the dynamics, assumption that actions affect both reward state and observations. But in reality there is usually a model for the rewards, some sort of assumption on the state dynamics, in some case observations are not affected by actions. Apr 16 '20 at 10:13

The term control comes from dynamical systems theory, specifically, optimal control. As Richard Sutton writes in the 1.7 Early History of Reinforcement Learning section of his book [1]

Connections between optimal control and dynamic programming, on the one hand, and learning, on the other, were slow to be recognized. We cannot be sure about what accounted for this separation, but its main cause was likely the separation between the disciplines involved and their different goals.

He even goes on to write

We consider all of the work in optimal control also to be, in a sense, work in reinforcement learning. We define a reinforcement learning method as any effective way of solving reinforcement learning problems, and it is now clear that these problems are closely related to optimal control problems, particularly stochastic optimal control problems such as those formulated as MDPs. Accordingly, we must consider the solution methods of optimal control, such as dynamic programming, also to be reinforcement learning methods.

Prediction is described as the computation of $$v_\pi(s)$$ and $$q_\pi(s, a)$$ for a fixed arbitrary policy $$\pi$$, where

• $$v_\pi(s)$$ is the value of a state $$s$$ under policy $$\pi$$, given a set of episodes obtained by following $$\pi$$ and passing through $$s$$.
• $$q_\pi(s, a)$$ is the action-value for a state-action pair $$(s, a)$$. It's the expected return when starting in state $$s$$, taking action $$a$$, and thereafter following policy $$\pi$$.

Control is described as approximating optimal policies. When doing control, one maintains both an approximate policy and an approximate value function. The value function is repeatedly altered to more closely approximate the value function for the current policy, and the policy is repeatedly improved with respect to the current value function. This is the idea of generalised policy iteration (GPI). See 5.1 Monte Carlo Control in [1].

[1] Reinforcement Learning: An Introduction, by Richard S. Sutton and Andrew G. Barto