Setting hyper-parameters for Deep Q-learning

Question

I would like to seek views on the appropriate hyper-parameters for an implementation of Deep Q-learning. Thanks for your help.

I’m following the implementation from the 2015 paper Deep Reinforcement Learning with Double Q-learning by Hasselt, Guez and Silver.

In fact not entirely because I am implementing a) Experience Replay and b) fix network for target Q-value, but I am not implementing c) clipping the Q-values at [-1,1].

The main hyper-parameters are epsilon schedule, size of experience replay buffer, batch-size, number of iterations for which the target network is kept fixed.

I am using as follows:

Epsilon: reduces from 1.0 to 0.0 over 100 iterations.

Size of experience replay buffer: 2,000.

Batch size: 500.

Number of iterations for which the target network is fixed: 3600.

I don’t seem to be getting great convergence. Would you be able to advise if these parameters are reasonable. And which one should I vary which is going to make the most meaningful difference for improving convergence? Also I wonder if any of the underlying parameters or design characteristics should be considered too.

For brief background:

• the episodes are of fixed size (36 steps, ie monthly projection over term horizon of three years) i.e. I am updating the target network every 100 episodes.

• the states are defined with three variables (holding in a particular asset, time to maturity, price of the asset)

• the network is two fully connected hidden layers with 512 units each, three inputs (state space dimension) and 20 outputs (20 actions possible in each state).

Thank you

Answer 1

The main problem with hyperparameter choices is that is very hard to tell what should work for any specific problem. You should try a range of each value, with big jumps (factors of 2 to 5) to search multiple possibilities.

That said, I think the following could be a problem:

Epsilon: reduces from 1.0 to 0.0 over 100 iterations.

This decay is too fast, and too extreme. In Q learning you should always have some small epsilon value whilst learning and set to zero only for tests. With a zero epsilon you stop exploring behaviour, and that significantly reduces the chance that the agent will find better actions than the ones it already has already decided on as "best".

I would suggest reduces from 1.0 to 0.05 over 100,000 iterations, remaining at 0.05 after that.

A rule of thumb I use for myself is that there should always be some reasonable but small probability of completing a trajectory taking only greedy actions, but that the "typcial" trajectory may have one or two experimental actions in expectation. So in your case with fixed episode length of 36, a minimum $\epsilon = 0.05$ would be ok - it would make $0.96^{36} \approx 0.16$ of your trajectories greedy with respect to current Q values, and the expected number of exploratory actions is 1.8.

The following may affect stability:

Size of experience replay buffer: 2,000.

I would pick maybe 10,000 or 100,000 initially. Q learning can still learn from older off-policy data, and there is some benefit from keeping older, worse experiences around to address catastrophic forgetting.

Batch size: 500.

This should probably be smaller. It should definitely be small compared to the replay memory, otherwise you risk catasrophic forgetting by learning with a huge bias towards recent experiences only.

Perhaps start at 25 or 50, and experiment with increasing it later on. Larger numbers are good for reducing variance if your learning data is has lots of noise, but try to keep the value under 1% of replay memeory size.