1
$\begingroup$

I am dealing with a process arising in time series. As an inspiration I refer to a paper by Mnih et al. Playing Atari with Deep Reinforcement Learning: https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf

Question 1: Am I following a general logic correctly?

  1. Accumulate a replay buffer size N with sequences of state-action-reward-next_state* agent's behaviour from let's say ealier part of my time series.

    • I use several successive sequences s-a-r until I hit a non-zero reward as one memory example, and I hardcoded the length of the sequences to allow easy passage to a neural network.

  2. Select random sparse sample of size M from the buffer to teach the network. Split M into minibatches.

  3. Initialize a new network if it did not exists, or use the network from previous time step (it's weight status) find max(Q) for next_state-actions. Correct target by reward + gamma * max(Q).

  4. Make mini-batch learning passing the updated target to the network.

  5. Make an estimate of an action, following e-greedy policy, for currently observed state; save the updated network.

  6. Make action in an emulator, observe reward (maybe zero == no reward), and next state. Save the new sequence to a random row in replay buffer when reward is non-zero (meaning, the sequence of state-actions riched its target).

Question_2: is it OK that my rewards can be negatively biased in the initial replay buffer?

Question_3: should I expect increasing max(Q) predicted over time as the network converges?

Question_4: I am not sure if I should use normalization (scaling) of max(Q) OR reward + max(Q) as an output for a relu-NN.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Question 1

1) You initialize your network here and use it to dump the memory replay.

2) What is random sparse sample here? Your memory replay consists of N elements. You just sample randomly K of them, where K is the minibatch size.

3) correct, but the net initialization should happen on the step 1).

4) If you do training on the previous step, why do you want to do the update again?

5) okay

6) Several issues here. Why do you want to save a transition in a random place of the memory replay? You will sample it randomly anyway, why do you want to do this?

You save the transition in the memory replay even if your reward is zero.

Question 2

 What do you mean by rewards being negatively biased?

Question 3

Sure. You should expect that. What does you Q function give you? It gives you the future discounted reward flow, right? The better you play, the more reward you should get.

Question 4

Not sure, that I get it correctly, can you elaborate more on that?

I recommend implementing the algorithm for some simple gym environment, and then adjusting it for your specific case. If you have any questions, I will be glad to help.

Improve this answer
$\endgroup$
9
  • $\begingroup$ Thanks for answering my long post. "You just sample randomly K of them, where K is the minibatch size." I thought I would select K, which will in turn be splitted in minibatches of size K / m (and I do m passes of NN update). "If you do training on the previous step, why do you want to do the update again?" One step is to calculate updated target, next step is to use the targets to train NN (all happening in one iteration). "Why do you want to save a transition in a random place of the memory replay?" To keep my buffer filled with most recent examples, while leaving a room for old $\endgroup$
    – Alexey Burnakov
    Aug 17, 2017 at 12:27
  • $\begingroup$ Not sure, that I get it correctly, can you elaborate more on that? I mean the relu function will be zero while support is < 1, and it will only became larger then zero when the input is >= 1. Moreover, what if my corrected targets are negative? ReLu does not output negative values. I though it would have made the NN converge worse, as opposed to (gamma * max(Q) + R) being in [0;1]. You think it could help? $\endgroup$
    – Alexey Burnakov
    Aug 17, 2017 at 12:31
  • $\begingroup$ "I recommend implementing the algorithm for some simple gym environment, and then adjusting it for your specific case. If you have any questions, I will be glad to help." My task at hand is very specific. It is already simplified and is a toy problem. I have a time series and I want to apply actions to it as the time passes so that I always guess right the direction of time series increments. I can have for example the following real-life sequence: s1-a1-0-s* -> s2-a2-0-s* -> s3-a3-0-s* -> s4-a4-0-s* -> s5-a5-+1. So I have to make a series of actions until hitting a target. $\endgroup$
    – Alexey Burnakov
    Aug 17, 2017 at 12:33
  • 1
    $\begingroup$ To keep later updates and removing the old ones you just pop the oldest example and push the newest when your memory replay reaches its limits. Concerning your negative rewards. You can use tanh instead of ReLU. I have never tried negative rewards, and am not aware of potential difficulties you might face though. $\endgroup$
    – yobibyte
    Aug 18, 2017 at 7:12
  • 1
    $\begingroup$ I do suppose, that one minibatch per step is better since you use data for newer policies for your future updates. With other words, you use too much of the data for more suboptimal policy when use train the net on several minibatches in a row. But this is not such an easy question, I suppose. Try to read about prioritized replay, may be it will help you. $\endgroup$
    – yobibyte
    Aug 18, 2017 at 14:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.