I am currently learning many aspects of Reinforcement Learning (RL), as is evident in many of my previous posts:

Reinforcement learning with subgoals

Model free reinforcement learning with subgoals: how to reinforce learning with only one reward?

Formulation of states for this RL problem and other questions.

This question is more 'soft', in it I am not really expecting a mathematically rigorous answer. It may bent on some frustrations I have while studying RL and some related topics.

So, I am using RL to model an interesting behavior found in some animals. (Sorry, this is as specific as I could go.)

In this kind of species, and in RL, there is a task to be done. This is simple, because there REALLY needs to be an objective, or a task to be performed. Otherwise, there wouldn't be a problem!

Now, here is the interesting part (plot twist): Something happens while this species perform the task. It exhibits a certain behavior BEFORE achieving the task. You can think of it as some preliminary task it should perform before the main task/goal can be achieved. This is the reason why all those questions on subgoals are present above.


I made a RL model of the task. I defined the state representation to be the relative position between the current position and the subgoal position. Rewards are given when the subgoal position is approached. Rewards are also given when the main task/objective is achieved.

Is it any good?

It gets the job done. You see it perform and whatever needs to be done has been achieved. However, I feel that the 'subgoal' part steals the limelight from the main task. Perhaps at this point, all I have done is show that whatever I have accomplished 'mimics' the interesting animal behavior that is supposed to be modeled.

What should be done, ideally?

I envision that the main goal should be in the state space representation and not just the subgoal. The main goal should play a more important role, I reckon. (And by 'more important', I suspect it is difficult to quantify this very subjective word.)

My main question

How do you include such a subgoal? Or ... How do you model the task using RL taking into account the interesting behavior I mentioned above?

  • 1
    $\begingroup$ I'd love to help you, but I'm honestly lost. What is it about your current formulation that dissatisfies you? That the reward function is primarily based on the subgoal, rather than the goal? $\endgroup$ Aug 25, 2016 at 19:53
  • $\begingroup$ Hi. Yes, the main dissatisfaction is that the state representation is based on the subgoal. This puts the subgoal in the spotlight, the star of the problem, when in fact it should have been the main goal which is the important. Furthermore, an ideal formulation (for me) would be to make the agent go/perform to the main goal, AND ALONG THE WAY make it perform the subgoals, without explicitly telling it to do so. $\endgroup$
    – cgo
    Aug 27, 2016 at 7:16


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.