I've been studying hierachial reinforcement learning problems, and while a lot of papers propose algorithms for learning a policy, they all seem to assume they know in advance a graph structure describing the hierarchy of the actions in the domain. For example, The MAXQ Method for Hierarchial Reinforcement Learning by Dietterich describes a graph of actions and sub-tasks for a simple Taxi domain, but not how this graph was discovered. How would you learn the hierarchy of this graph, and not just the policy?

In other words, using the paper's example, if a Taxi were driving around aimlessly, with little prior knowledge of the world, and only the primitive move-left/move-right/etc actions to take, how would it learn higher level actions like go-pick-up-passenger? If I'm understanding the paper correctly (and I may not be), it proposes how to update the policy for these high-level actions, but not how they're formed to begin with.


According to this paper

In the current state-of-the-art, the designer of an RL system typically uses prior knowledge about the task to add a specific set of options to the set of primitive actions available to the agent.

Also see section 6.2 Learning Task Hierarchies in the same paper.

The first idea that comes to my mind is that if you don't know task hierarchies, you should start with non-hierachial reinforcement learning and trying to discover the structure afterwards or while learning, i.e. you are trying to generalize your model. To me this task looks similar to Bayesian model merging technique for HMM (for example see this thesis)

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.