I am currently learning many aspects of Reinforcement Learning (RL), as is evident in many of my previous posts:
This question is more 'soft', in it I am not really expecting a mathematically rigorous answer. It may bent on some frustrations I have while studying RL and some related topics.
So, I am using RL to model an interesting behavior found in some animals. (Sorry, this is as specific as I could go.)
In this kind of species, and in RL, there is a task to be done. This is simple, because there REALLY needs to be an objective, or a task to be performed. Otherwise, there wouldn't be a problem!
Now, here is the interesting part (plot twist): Something happens while this species perform the task. It exhibits a certain behavior BEFORE achieving the task. You can think of it as some preliminary task it should perform before the main task/goal can be achieved. This is the reason why all those questions on subgoals are present above.
I made a RL model of the task. I defined the state representation to be the relative position between the current position and the subgoal position. Rewards are given when the subgoal position is approached. Rewards are also given when the main task/objective is achieved.
Is it any good?
It gets the job done. You see it perform and whatever needs to be done has been achieved. However, I feel that the 'subgoal' part steals the limelight from the main task. Perhaps at this point, all I have done is show that whatever I have accomplished 'mimics' the interesting animal behavior that is supposed to be modeled.
What should be done, ideally?
I envision that the main goal should be in the state space representation and not just the subgoal. The main goal should play a more important role, I reckon. (And by 'more important', I suspect it is difficult to quantify this very subjective word.)
My main question
How do you include such a subgoal? Or ... How do you model the task using RL taking into account the interesting behavior I mentioned above?