I'm new in deep q-learning and I have understood the main concepts of it and I'm trying to solve problems with DQL. The problem is that I don't know how to initialize some key values of the algorithm and the deep q network?! Here's my main questions:

  • What is the best value for discount factor? If it depends on problem, how can you define it?
  • How can you define the best learning rate?

And about the DQN (Deep Q-Network):

  • Normally how many layers are needed? Again, if it depends on problem, what is the best practice?
  • What initializer function is more proper for weights initialization?
  • Do we have to use mean squared error as loss function? If it's not, what is alternatives and when to use them?
  • Is it normal if loss value increases all the time? (absolutely I'm talking about q-learning, not other situations. And the reason of this question is that I had a working DQN that it's loss value increased all the time! but the algorithm works pretty well!)

And finally, how can you find out how many steps are required for learning in partially observable problems? (problems that we don't have the entire of it's environment)

Thanks in advance.

  • $\begingroup$ I think this is too many questions in one go. It makes it too much work to answer. I suggest only asking the "main questions" this time around, and asking the other parts in separate questions as you progress. $\endgroup$ – Neil Slater Jan 18 at 8:31

Yes, these are all part of hyperparameter selection and optimization, but there are some good rules of thumb.

Discount factor: Somewhere between 0.9 and 0.99999. The sparser and farther away rewards are, the higher you probably want this. Learning rate: between 0.1 and 0.00001. Layers: At most 3 in most cases. Initializer: you can't really go wrong as long as you choose a reasonably well known initialization. Loss: no, you could also use L1 loss or any other regression loss. Number of steps for POMDP: use domain expertise, ask yourself how many steps of an environment it would take you to infer most of the hidden state.

| cite | improve this answer | |
  • $\begingroup$ Thanks for your answer. It was very helpful and I up-voted it, but unfortunately it's not completely what I'm looking for. $\endgroup$ – Hamidreza Feb 2 at 13:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.