I've been studying hierachial reinforcement learning problems, and while a lot of papers propose algorithms for learning a policy, they all seem to assume they know in advance a graph structure describing the hierarchy of the actions in the domain. For example, The MAXQ Method for Hierarchial Reinforcement Learning by Dietterich describes a graph of actions and sub-tasks for a simple Taxi domain, but not how this graph was discovered. How would you learn the hierarchy of this graph, and not just the policy?
In other words, using the paper's example, if a Taxi were driving around aimlessly, with little prior knowledge of the world, and only the primitive move-left/move-right/etc actions to take, how would it learn higher level actions like go-pick-up-passenger? If I'm understanding the paper correctly (and I may not be), it proposes how to update the policy for these high-level actions, but not how they're formed to begin with.