# Reinforcement Learning on Historical Data

I have been working on learning the optimal policy of communications for customers (which notifications to send, how many to send and when to send). I have historical data of past notifications sent (with timestamps) and their performances. Was trying to apply RL to this problem in order to learn the optimal policy. However, one key constraint here is that I do not have the luxury of learning the policy on the fly (on-line) as I do not currently control the actions (which notifications can be sent to what customers). I have two questions:

1. Is RL the right framework under such constraints?
2. How can we learn the optimal policy offline in such situations and how do we evaluate the same?

1. Is RL the right framework under such constraints?

It looks possible, but maybe some small detail that you have not given would make other approaches more feasible. For instance, if the notification events can be treated as more or less independent, then a supervised learning approach might be better, or at least more pragmatic.

More practically it is not 100% clear what your state, timesteps and action choices will be. These need to be well-defined for RL approaches to work. In addition, you want to be able construct states that have (or nearly have) the Markov property - essentially that anything known and non-random about expected reward and next state is covered by the state.

1. How can we learn the optimal policy offline in such situations

You want both an offline (data is historical, not "live") and off-policy (data is generated by a different policy to the one you want to evaluate) learner. In addition, I am guessing that you don't know the behaviour policies that generated your data, so you cannot use importance sampling.

Probably you can use a Q-learning approach, and work through your existing data either by replaying each trajectory using Q($\lambda$) in batches, or some variant of DQN using sampled mini-batches.

This is not guaranteed to work, as off-policy learning tends to be less stable than on-policy, and may require several attempts to get hyper-parameters that will work. You will need a good number of samples that cover optimal or near optimal choices on each step (not necessarily in the same episodes), because Q-learning relies on bootstrapping - essentially copying value estimates from action choices backwards to earlier timesteps so as to influence which earlier states the agent prefers to take actions to head towards.

If your state/action space is small enough (when you fully enumerate the states and actions), you may prefer to use the tabular form of Q-learning as that has some guarantees of convergence. However, for most practical problems this is not really possible, so you will want to look at options for using approximation functions.

... and how do we evaluate the same?

If you can get realistic-looking converged action-values from your Q-learning (by inspection), then there are only 2 reasonable ways to assess performance:

• By running the agent in a simulation (and maybe further refining it there) - I don't expect this is feasible for your scenario, because your environment includes decisions made by your customers. However, this is a good stepping-stone for some scenarios, for instance if the environment is dominated by basic real-world physics.

• By running the agent for real, maybe on some subset of the workload, and comparing actual rewards to predicted ones over enough time to establish statistical confidence.

You could also dry-run the agent alongside an existing operator, and get feedback on whether its suggestions for actions (and predictions of reward) seem realistic. That will be subjective feedback, and hard to assess performance numerically when the actions may or may not be used. However, it would give you a little bit of QA.

Now you already have the historical action and performance, this is a classical supervised learning problem that maps your (customer profile, action) tuple to a performance score.

1. Reinforcement learning makes very INEFFICIENT use of data, so usually it requires kind of infinite amount of supplied data either from simulator or real experience. I would think neither of these cases apply to you, since you will not want your untrained model to send random notifications to your customers at the beginning state of training, and your problem will be considered solved if you already have a simulator.

2. Reinforcement learning is usually used to deal with long sequences of actions, and the early action could have drastic influence on the final outcome, such as in chess. In that case, there is no clear partition of the final reward received at the end to each step of your actions, hence the Bellman equation is used explicitly or implicitly in reinforcement learning to solve this reward attribution problem. On the other hand, your problem does not seem to have this sequential nature (unless I misunderstood or your system is emailing back and forth with a customer), and each sample from your data is a single-step IID.

These papers provides a method called Fitted Q Iteration for batch reinforcement learning (i.e learning a policy from past experiences) https://pdfs.semanticscholar.org/2820/01869bd502c7917db8b32b75593addfbbc68.pdf https://pdfs.semanticscholar.org/03fd/37aba0c900e232550cf8cc7f66e9465fae94.pdf

You will need a clearly defined reward function, states and actions.

For testing, the best is to use a small user cohort and A/B test with regards to your metrics.