Reinforcement learning for continuous states, discrete actions. Algorithms?

Question

I'm trying to find optimal policy in environment with continuous states (dim. = 20) and discrete actions (3 possible actions). And there is a specific moment: for optimal policy one action (call it "action 0") should be chosen much more frequently than other two (~100 times more often; this two action more risky).

I've tried Q-learning with NN value-function approximation. Results were rather bad: NN learns always choose "action 0". I think that policy gradient methods (on NN weights) may help, but I don't understand how to use them on discrete actions.

Could you give some advise what to try? (maybe algorithms, papers to read). What are the state-of-the-art RL algorithms when state space is continuous and action space is discrete?

Thanks.

Answer 1

You could use SARSA with function approximation to handle the continuous states. A good reference is the "Reinforcement learning: An introduction" by Sutton and Barto. Q-learning with function approximation is not proven to converge (although it might work in some specific cases). SARSA with linear function approximation is proven to converge to a ball around the optimal point for MDP's so with this approach you'll have some guarantees. For your function approximation you could also use neural networks. In any case, as I mentioned before, Sutton's book is a good reference