5
$\begingroup$

I'm implementing DQN in reinforcement learning, double DQN actually, and it's returning only one action, out of 7 possible actions. I've tried changing number of layers of neural network, regularization parameters, layer size. But I still get only one action. That is, for a particular set of parameters, I get only one action. I wanted to know if it is a common scenario. What it would imply and under what conditions we will encounter such situation? Any suggestion for resolving this will be greatly appreciated.

$\endgroup$

1 Answer 1

2
$\begingroup$

It varies on how you implemented everything(please show us the code), but these are some possibilities:

  • a bug in the program
  • initial weights are not so good
  • state-action values are infinity/NaN

I think you should check your state-action values and verify that they're not diverging to infinity/NaN, because that's most likely the case.

$\endgroup$
5
  • $\begingroup$ As you suggested, Q values were very high... 10^22! When I searched for reason, i realized that i had one input value which was very high, and it caused very high Q values, in turn resulting only one action. Once I removed that input (which was strictly not needed anyway), I got the things running smoothly with different actions as output. Your suggestions indeed helped!! $\endgroup$ Commented Aug 25, 2017 at 8:10
  • $\begingroup$ @StatMan Awesome to know that helped! $\endgroup$
    – nedward
    Commented Aug 25, 2017 at 8:11
  • $\begingroup$ Just out of curiosity, how did you come up with these suggestions? $\endgroup$ Commented Aug 25, 2017 at 9:20
  • 1
    $\begingroup$ @StatMan I've recently had the exact same problem. Turns out that DQN, or Q Learning in general tends to diverge when the function approximation is non-linear, as in most cases. I guess experience pays off! $\endgroup$
    – nedward
    Commented Aug 25, 2017 at 10:37
  • $\begingroup$ Learning by Experience pays off not just for machines but humans as well. Amen! $\endgroup$ Commented Aug 25, 2017 at 13:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.