I'm trying to find optimal policy in environment with continuous states (dim. = 20) and discrete actions (3 possible actions). And there is a specific moment: for optimal policy one action (call it "action 0") should be chosen much more frequently than other two (~100 times more often; this two action more risky).

I've tried Q-learning with NN value-function approximation. Results were rather bad: NN learns always choose "action 0". I think that policy gradient methods (on NN weights) may help, but I don't understand how to use them on discrete actions.

Could you give some advise what to try? (maybe algorithms, papers to read). What are the state-of-the-art RL algorithms when state space is continuous and action space is discrete?


  • $\begingroup$ An optimal policy would map every observation to the action that yields maximum expected reward in the long run. Meaning, specific states are mapped to specific actions, not distributions of actions, where some are taken merely more often than others. $\endgroup$
    – Don Reba
    Nov 19, 2014 at 9:49
  • $\begingroup$ I don't get how it can help. Please, explain again. $\endgroup$
    – alex_io
    Nov 19, 2014 at 13:11

1 Answer 1


You could use SARSA with function approximation to handle the continuous states. A good reference is the "Reinforcement learning: An introduction" by Sutton and Barto. Q-learning with function approximation is not proven to converge (although it might work in some specific cases). SARSA with linear function approximation is proven to converge to a ball around the optimal point for MDP's so with this approach you'll have some guarantees. For your function approximation you could also use neural networks. In any case, as I mentioned before, Sutton's book is a good reference

  • $\begingroup$ Q-learning with linear function approximation will also converge. $\endgroup$
    – A.D
    Dec 4, 2017 at 10:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.