3
$\begingroup$

Suppose I have collected some data, where each datum is an interaction of an agent with an environment. From this, I then train a neural network to learn a dynamics model, and to predict the next state $s_{t+1}$ given state $s_t$ and action $u_t$.

What I want to do now is, for any given initial state $s_0$, predict the optimum sequence of actions $u_t, ..., u_{t+n}$ that will cause my agent to end up in state $s_{goal}$.

How can I do this?

I'm not very familiar with this field, but one idea is to formulate it as a Markov Decision Process (MDP). I would give a positive reward depending on the final distance to the goal, and small negative reward for each action, to encourage the agent to reach the goal quickly. I would then solve the MDP using Value Iteration. Would this make sense? Or does this only apply for discrete states? (My state and action spaces are continuous.)

Or are there any other ways to solve this?

Thanks.

$\endgroup$

1 Answer 1

3
$\begingroup$

Predicting the next state is not the usual starting point in reinforcement learning*. Instead there are two main approaches:

  • Predict future cumulative reward from a given state or state/action pair (called "return" or "utility"), and choose actions that maximise this quantity. Sometimes "advantage" is used as the quantity, which is the difference between average expected return and that for the given action.

  • Predict the action that will produce the best future accumulated reward directly, and adjust the predictions depending on results of experience.

The two approaches are often combined in "Actor-Critic" methods, and this would be a reasonable approach when you have continuous action space.

I would give a positive reward depending on the final distance to the goal, and small negative reward for each action, to encourage the agent to reach the goal quickly.

Generally with reinforcement learning you should avoid using heuristics and only provide rewards that measure the goal. So a positive reward for reaching the goal (or if the task is episodic, then reward based on final distance from the goal), and a negative reward per time step would be my suggestion. One way of framing state values in RL is that they are learnable heuristics - you set up the problem with rewards matching goals, and the agent learns local heuristics from experience of its attempts to reach the goal.

This article on the cutting edge AC3 algorithm might help you get started - it may jump in a bit deep if you want to learn the underlying theory. But it shows an implementation that you could make use of, and if you have time you could work from the start of the tutorial series, which walks through the basics first (using TensorFlow).


* It can be a useful addition to have a next-state predictor, as demonstrated in part 3 of the same tutorial series on Medium. It can also be used for planning moves several steps in advance. However, it is not the primary driver of RL agents.

$\endgroup$
3
  • $\begingroup$ Hi Neil, thanks very much for your helpful reply! The reason I thought I should train a network to predict the next state (learn a dynamics model of the environment), is that the particular task is not defined until test time. Therefore, I cannot really assign a reward function and train a network to predict future reward, or the action which will produce the best future reward. Imagine the task being one where the agent is in a big room, and is assigned a random goal to reach. There would be an infinite number of tasks/rewards. Instead, could I somehow query the dynamics model at test time? $\endgroup$ Oct 11, 2017 at 15:41
  • 1
    $\begingroup$ @Karnivaurus: That can be resolved by making the immediate goal part of the state (or altering state values to be relative to the goal). For instance, if your goal is to have the agent navigate between a number of waypoints, which you only know at test time, have the state include information about waypoints and train using suitable random distribution of them, and +R for each waypoint reached. The last paper I linked had references to re-using the training in unseen test scenarios, your intuition might be on to something! But if you are working on a simpler problem, it may not be needed. $\endgroup$ Oct 11, 2017 at 15:45
  • $\begingroup$ @Karnivaurus: I have updated the answer, as using next-state predictions is more common than I realised. Still not the starting point of RL approaches though $\endgroup$ Oct 17, 2017 at 18:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.