I've written a Double DQN-based stock trading bot using mainly time series stock data.

I've recently upgraded my Experience Replay(ER) code with a version of Prioritized Experience Replay (PER) similar to the one written by OpenAI. My DQN's reward function is the stock return over 30 days (the length of my test window).

The strange thing is, once the bot has been trained using the same set of time series data and let free to trade on unseen stock data, the version that uses PER actually comes up with worse stock returns than the version using a regular ER.

This is not quite what I'd expected but it's very hard to debug and see what might have gone wrong.

So my question is, will PER always perform better than a regular ER? If not, when/why not?


1 Answer 1


Prioritization introduces bias. In typical applications, this is actually desired, but it might work against you in your specific environment.

An important thing that prioritization does for you is that it tends to favor transitions in which a non-zero reward was observed. This is helpful in environments like the atari games, in which rewards are relatively sparse (most transitions have $r=0$). If your environment has a denser reward structure, this kind of prioritization may not be beneficial.

Other than that, it's hard to make any definite claims without knowing more about the specific environment.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.