Reinforcement learning with subgoals The usual reinforcement learning task is that an agent starts from a start position with the ultimate aim to reach a goal. More often than not, RL algorithms involve planning and learning optimal policies to reach this goal and rewards are only awarded when the goal is attained. Sometimes, there is a penalty for each time step when the goal is not reached. 
Now, I am interested in a reinforcement learning task with a start and goal, but with one additional requirement: there are some subgoals that need to be achieved. 
The subgoals will have to be achieved first (and perhaps in a certain order) before the actual goal will be achieved. 
I want to ask: 


*

*If this is a totally new field/area, or a modification of the existing RL that I know. 

*What added conditions can be put in place to include the "subgoals" as specified? 

*Please send some references. 

 A: Sure, standard MDPs can hack that. You just need to carefully define the state space so that it includes all possible states, and ensure that the transition function reflects that order.
Consider a simple, square grid with length three on each side. The agent begins at the bottom left. The reward for the top-right—where the Master Sword resides—is some positive constant; other rewards are uniformly $0$.
There's little to this problem. You can be in any of nine states, and can transition to each state from any of its adjacent states. You can index them by an integer, or row and column. Row and column is intuitive: You can quickly tell from the row and column indices of two states whether a move between the two is legal. So, we'll define each state as a tuple $(i,j)$. Value iteration, or any planning algorithm, would make short work of it. 
Let's add the possibility that the top-right is not a terminal state, and that the agent can move about after retrieving the sword. Changing the state space is easy enough: States are now $(i,j,m)$, $m$ being a boolean indicating whether the sword has been retrieved. (If it helps, you can think of this as two grids, one corresponding to having the sword, one not.)
Our transition function also needs an upgrade, since our agent will never drop the sword. (It's very precious.) Easy enough: $T(s,a,s') = 0$ for any $s = (i_0,j_0,1), s' = (i_1,j_1,0)$.
Now, say that in each square in the diagonal from the top-left to bottom-right are the pendants of Courage, Power and Wisdom, respectively, and that the agent must collect these, in order, prior to retrieving the Master Sword. The new states are $(i,j,n)$, $n$ being the number of artifacts (pendants & sword) retrieved. 
You can visualize this state space as a stack of grids; gathering the next pendant or sword is loosely like 'moving up' a level. (We're up to 45 states total.) Your transition function should again account for this, returning $0$ for any transition that would represent giving up an artifact. Value iteration would propagate value out and down from the Master Sword.
If unordered gathering is allowed, we'll need more states still. Before, having the Pendant of Power implied having the Pendant of Courage. Since that's no longer the case, we'll need a boolean for each pendant and the Master Sword; states are now $(i,j,c,p,w,m)$ tuples, with any state with $m=1$ having $c = p = w = 1$.
Important point being, your state space is, intuitively, just a list of unique, exclusive conditions. This representation can capture order, once you put enough thought into your problem's full realm of possible states, and adjust your transition function accordingly.
A: Research has been targeted at RL with subgoals. It is sometimes referred to as hierarchical reinforcement learning [0]. Most works formalize this using the options framework [1]. For example environments, see these [2] [3].
A thing that is relatively recent is enforcing the subgoals somehow.
[0] https://www.youtube.com/watch?v=e8b0yC6COJ8
[1] Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. RS Sutton, D Precup, S Singh. 1999
[2] Building Portable Options: Skill Transfer in Reinforcement Learning. GD Konidaris, AG Barto. 2007
[3] Modular multitask reinforcement learning with policy sketches. J Andreas, D Klein, S Levine. 2017
