# Is there any way to make the notion of a “policy” in reinforcement learning less abstract?

In many reinforcement learning related literature, I see the author suddenly introduces an abstract function called policy $$\pi$$ which maps from the state to actions.

In other words, $$\pi$$ is a function, such that $$\pi: s \mapsto a$$.

Because the domain and range of this function are usually discrete, therefore this function do not have the usual properties when it comes to functions, such as continuity, differentiability, Lipischitzness, smoothness, etc. Therefore there is nothing general that we can say about this function, am I correct?

To make matters worse, often authors will freely use the term "expert policy", $$\pi^{\text{expert}}$$. The expert here usually being a human, so this expert policy is literally a function that models how the human brain works or react to a highly complicated environment. I am pretty sure there is no closed form description of such an object. There is virtually nothing we can ever say about this function either.

Is there any way or any existing literature that attempts to make the concept of a policy less abstract? What is the simplest policy that you can have for a non-trivial scenario?

Overall, I find it difficult to appreciate this concept and see how theory can connect to practice if I don't see a closed-form examples of a policy.

• flying an autonomous quadcopter is a popular example of a continuous space: arxiv.org/pdf/1709.03339.pdf . Continuous action spaces are less common but are studied: proceedings.mlr.press/v80/lee18b/lee18b.pdf – olooney Jun 11 '19 at 1:26
• See "Reinforcement Learning: An Introduction" by Sutton and Barto for an explanation :). – Chris Jun 11 '19 at 1:41
• Not sure if you knew this but policy $\pi$ is a probability distribution over action space for a specific state $s$. There is nothing abstract about it, it doesn't map from states to actions, it maps from state-action pair to a probability of taking a specific action $a$ in state $s$, more clearly, $\pi (a | s)$ is a probability of taking action $a$ while being in state $s$. – Brale_ Jun 11 '19 at 6:13

A real world example of policy are the house rules that Black Jack dealers adhere to. The dealer is the agent. The rules, which describe when the dealer can and cannot play cards, are the policy. The state is defined by the cards the dealer and the player(s) have. The possible actions include whether to play a card or not. The reward is winning or losing the hand.

This link talks about black jack from the dealer's perspective and some of the rules they play by. Dealer's perspective

There are two points to using continuous functions. One is that this is a way to manage continuous domains. If your potential actions are continuous, trying to discretize it will make RL unusable for that problem. The second is that using continuous functions allows the application of powerful mathematical tools such as gradient descent.

The point isn't that the features in the probability functions are related to each other on some n-dimensional surface. The point is that a surface that fits the probability function lends itself to analysis and use better then a table of numbers would.

• Nice example! Welcome to the side @jeffry7. – Matthew Drury Jun 23 '19 at 21:34

As Brale mentions, it's actually a function from states to a distribution over the set of actions.

Because the domain and range of this function are usually discrete,

The state is very often continuous in many control problems where you have continuous joint angles and pose, etc. The action space is also often continuous in those cases, since you can often choose to apply any amount of torque within some limits.

therefore this function do not have the usual properties when it comes to functions, such as continuity, differentiability, Lipischitzness, smoothness, etc. Therefore there is nothing general that we can say about this function, am I correct?

Often, for continuous spaces, $$\pi(s) = \mathcal{N}(f(s;\theta), \sigma^2)$$ where $$f(s;\theta)$$ is a learned function approximator. You can actually say that

1. $$f$$ is continuous, and subdifferentiable, if it is implemented as a neural network
2. $$f$$ is Lipschitz if there is a bound on the absolute value of the weights in $$\theta$$, alternatively you can enforce this with a weight regularization penalty a la WGAN-GP.

The expert here usually being a human, so this expert policy is literally a function that models how the human brain works or react to a highly complicated environment. I am pretty sure there is no closed form description of such an object. There is virtually nothing we can ever say about this function either.

That's true, but usually there is no need to make any such assumptions.

Is there any way or any existing literature that attempts to make the concept of a policy less abstract? What is the simplest policy that you can have for a non-trivial scenario?

A simple but powerful class of policies might be parameterized as $$\pi(s) = \mathcal{N}(\theta^T \phi(s), \sigma^2)$$ where $$\phi$$ is a set of basis functions.

• I am really interested in your example $\pi(s) = \mathcal{N}(f(s;\theta), \sigma^2)$. Do you know where I might encounter such a policy? The thing with me is that I have never seen a policy explicitly written down in closed form. – Cauchy's Carrot Jun 24 '19 at 18:11
• @SquaringtheCircleisEasy well you rarely see it written out like that because people say "linear policy" and assume it is clear what's meant. But here are some articles which use it. arxiv.org/abs/1803.07055 arxiv.org/abs/1802.05054 arxiv.org/abs/1703.02660 – shimao Jun 25 '19 at 5:35

Mathematics is about expressing ideas even if they cannot be solved. Expressing an idea as a function (in this case a policy) allows us to write it down and move it around in a quantitative context. This (often) ends up allowing us to make some quantitative use of it.

It turns out that in this particular example a policy over a discrete action space can expressed as the probability of an action, giving us the ability to take derivatives. This (and the existence of natively continuous action spaces) is the basis of a sub-field in Reinforcement Learning in which policies are approximated by a function (recently, a neural network) directly.