How to define number of states in reinforcement learning

Question

I'm a robotic engineer who's relatively new to reinforcement learning and I want to try to do simple reinforcement learning on a robot to optimize its velocity. I am however having trouble with defining the states.

The robot always begins in its home state, then receives a random coordinate in the working space that it must move to. Using reinforcement learning, it must optimize its own motor settings so that the movement is executed smoothly, in other words it has to predict a parameter number setting. The RL predicts the optimal parameter, the robot then moves to the specified coordinate using this setting and then moves back to the home state, where it receives a reward for how well the total motion was executed.

Now I want to define this problem as a reinforcement learning problem (I eventually want to use actor-critic). The action here is the setting of that one parameter, or choosing a number. However, I'm confused about the number of states this problem has.

• My current guess is that there are two states: the home state coordinates and the new coordinates the robot moves to. It starts in the home state, takes an action of setting the motor parameter and then moves to the new coordinates with that setting. It then moves back to home, receives a reward and the episode ends. Next episode, it receives a new coordinate and repeats the process, eventually learning what parameter setting is optimal for which coordinate.
• However, what still confuses me is that the robot then moves back to the home state, where it receives the reward. So, the problem could also be seen as a 1 state problem, where the robot is in its home state, then executes a full back and forth motion with a certain parameter setting, and then receives a reward for that action.

Which of the two is right? I was planning to start with TD(0), where there's two states and one step, but I'm doubting if the problem as defined above even has two states. I would really appreciate someone shedding some light on this. Thanks in advance!

Answer 1

Your problem does not have time steps, but does have state in the form of a changing target location, which you expect to influence what the optimal action is. Given this, the problem more closely resembles a contextual bandit problem where you want to associate an ideal response to some variable input, and learn the association through experimentation by the agent.

Your state space is the space of possible target co-ordinates. The precise target location affects the score that you will get for any given input parameters. The origin position does not change, so does not impact action choice. This is probably far more than one or two states - it seems likely that this is a continuous state space, unless you have a set of fixed target locations. In RL terms you cannot even enumerate such states. So far more than one or two.

I was planning to start with TD(0), where there's two states and one step, but I'm doubting if the problem as defined above even has two states

I suggest you start with a gradient contextual bandit solver, perhaps as described in Contextual Bandits with Continuous Actions: Smoothing, Zooming, and Adapting if your action space is also continuous.

You may want to start with a simpler problem definition with discrete states and discrete actions first to practice RL concepts.

If your long-term goal is to provide control where there are time steps with interim states and actions during an episode, then you can also treat the whole thing like a 1-step MDP and use RL solvers, expanding to more steps once you have that working. Formulating as 1-step MPD would add 1 discrete state - ending the episode by terminating back at origin. Yo udon't need to learn the value of that state though, it will be 0 by definition.

Tabular TD(0) is out of the question though, because of the large state space. Instead, you would need to move immediately to something like DQN which uses neural networks to learn the action value function. Technicallly DQN with single-step is a specific implementation of TD learning - it adds a lot of detail, but at its core it generates TD targets and uses them for updates to a value function.

If action space is also continuous then DQN wont work either, and you will need a policy gradient or actor-critic approach like REINFORCE, A3C, DDPG. These are harder to comprehend at the theoretical level though, so again you may prefer to work on a few toy problems before tackling a control problem with continuous state and continuous action spaces - these are quite complex to work with in reinforcement learning. You might even find something like random search or genetic algorithm search for ideal parameters is sufficient, if your goal is to tune the parameters approximately, as opposed to study reinforcement learning.