I'm a robotic engineer who's relatively new to reinforcement learning and I want to try to do simple reinforcement learning on a robot to optimize its velocity. I am however having trouble with defining the states.
The robot always begins in its home state, then receives a random coordinate in the working space that it must move to. Using reinforcement learning, it must optimize its own motor settings so that the movement is executed smoothly, in other words it has to predict a parameter number setting. The RL predicts the optimal parameter, the robot then moves to the specified coordinate using this setting and then moves back to the home state, where it receives a reward for how well the total motion was executed.
Now I want to define this problem as a reinforcement learning problem (I eventually want to use actor-critic). The action here is the setting of that one parameter, or choosing a number. However, I'm confused about the number of states this problem has.
- My current guess is that there are two states: the home state coordinates and the new coordinates the robot moves to. It starts in the home state, takes an action of setting the motor parameter and then moves to the new coordinates with that setting. It then moves back to home, receives a reward and the episode ends. Next episode, it receives a new coordinate and repeats the process, eventually learning what parameter setting is optimal for which coordinate.
- However, what still confuses me is that the robot then moves back to the home state, where it receives the reward. So, the problem could also be seen as a 1 state problem, where the robot is in its home state, then executes a full back and forth motion with a certain parameter setting, and then receives a reward for that action.
Which of the two is right? I was planning to start with TD(0), where there's two states and one step, but I'm doubting if the problem as defined above even has two states. I would really appreciate someone shedding some light on this. Thanks in advance!