I would like to seek views on the appropriate hyper-parameters for an implementation of Deep Q-learning. Thanks for your help.
I’m following the implementation from the 2015 paper Deep Reinforcement Learning with Double Q-learning by Hasselt, Guez and Silver.
In fact not entirely because I am implementing a) Experience Replay and b) fix network for target Q-value, but I am not implementing c) clipping the Q-values at [-1,1].
The main hyper-parameters are epsilon schedule, size of experience replay buffer, batch-size, number of iterations for which the target network is kept fixed.
I am using as follows:
Epsilon: reduces from 1.0 to 0.0 over 100 iterations.
Size of experience replay buffer: 2,000.
Batch size: 500.
Number of iterations for which the target network is fixed: 3600.
I don’t seem to be getting great convergence. Would you be able to advise if these parameters are reasonable. And which one should I vary which is going to make the most meaningful difference for improving convergence? Also I wonder if any of the underlying parameters or design characteristics should be considered too.
For brief background:
the episodes are of fixed size (36 steps, ie monthly projection over term horizon of three years) i.e. I am updating the target network every 100 episodes.
the states are defined with three variables (holding in a particular asset, time to maturity, price of the asset)
the network is two fully connected hidden layers with 512 units each, three inputs (state space dimension) and 20 outputs (20 actions possible in each state).