2
$\begingroup$

I am interested in the problem of estimating the policy parameters of a behaving agent, in the usual reinforcement learning setup (i.e. the agent goal is to maximize his cumulative reward and has no prior knowledge about the environment's dynamics and the reward function).

While most literature is concerned with how to build this agent, I am interested in the in estimating the agent's policy by observing his behavior and interaction with the environment.

In other words, imagine we add to the reinforcement learning setup an "observer", looking at the agent interacting with the environment, and his goal is to model the agent behavior. Note that we can assume that the observer has full knowledge about the environment dynamics.

I assume that in the general case this problem is very difficult, however, under certain restrictions (assuming bandits or contextual bandits, small policy families, etc...), it may be computationally tractable.

I tried to look for literature dealing with this problem and couldn't find, can anyone directing me in finding the correct keywords, problem name, or anything that can help me find related work.

$\endgroup$
1
$\begingroup$

The term for this is "imitation learning" or "learning from demonstrations", and in fact is often used to train policies which learn from human agents on tasks where the search space is too large to make learning from scratch feasible.

It is possible to just train a supervised model which predicts the agent's action in the next time step in a very straightforward way.

$\endgroup$
  • $\begingroup$ thank you for your answer. Learning from demonstration assumes that the policy of the agent is stationary however I am interested in estimating the agent's current policy at each step while the agent is learning, i.e., improving/changing its policy. $\endgroup$ – Goek May 11 '18 at 5:24
  • $\begingroup$ @Goek why not train the supervised model "online" so that it is always an estimate of the current policy. $\endgroup$ – shimao May 11 '18 at 5:36
0
$\begingroup$

Below are two papers that target this question:

  1. Trial-by-trial data analysis using computational models
  2. Estimating Internal Variables of a Decision Maker’s Brain: A Model-Based Approach for Neuroscience

The following answer is based on the explanation given in the first reference.

To estimate the agent policy trail-by-trial, first we should assume some learning model $M$ (a learning process that the agent uses), for instance, one can assume Q learning. In this case, the model parameters are the learning rate $\alpha$ and temperature $\beta$.

By Bayes' rule we can estimate the posterior probability distribution of the parameters model $P(\theta_M|D,M)$ by the following: $P(\theta_M|D,M) \propto P(D|M,\theta_M) \cdot P(\theta_M |M)$. Since we are interested in $\hat \theta_M = \max_{\theta_M} P(\theta_M|D,M)$, the propotion is enough.

The first term can be calculated using the model specification and the second term is a prior that is determined by the knowledge about the model. Then the model parameter that best fit the data $\hat \theta_M$ can be calculated analytically for simple models by maximizing over the posterior probability with respect to $\theta_M$. If calculating the derivations with respect to $\theta$ is intractable, one can use Monte Carlo based methods. With the ability to calculate the data likelihood, we can draw any combination of possible $\theta_M=<\alpha, \beta>$ as select the combination with the highest likelihood.

For instance, assume we observe an agent in a 2-arms bendit setup {Right(R),Left(L)}, and that the learning model (in this case the action value) is updated as follows: $Q_{t+1}(c_t) = Q_t(c_t)+ \alpha (r_t-Q_t(c_t))$, and the action probability $P(c_t)$ taken according to the following: $P(c_t|Q_t(L), Q_t(R)) = \frac{\exp(\beta \cdot Q_t(L))}{\exp(\beta \cdot Q_t(L))+ exp(\beta \cdot Q_t(R))}$. Then, given the observed sequences $c_{1:T}$ and $r_{1:T}$, its likelihood can be calculated by: $P(D|M,\theta_M) = \Pi_t P(c_t|Q_t(L),Q_t(R))$. $Q_t$ is determined by the reward sequence $r_{1:T}$ and the equation for $Q_t$ given before.

Here it would be easier to use Monte Carlo Method to estimate $\hat \theta_M$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.