Neural Q Learning workflow and behaviour I am dealing with a process arising in time series. As an inspiration I refer to a paper by Mnih et al. Playing Atari with Deep Reinforcement Learning: https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf
Question 1: Am I following a general logic correctly?


*

*Accumulate a replay buffer size N with sequences of state-action-reward-next_state* agent's behaviour from let's say ealier part of my time series.


*

*I use several successive sequences s-a-r until I hit a non-zero reward as one memory example, and I hardcoded the length of the sequences to allow easy passage to a neural network.


*Select random sparse sample of size M from the buffer to teach the network. Split M into minibatches.

*Initialize a new network if it did not exists, or use the network from previous time step (it's weight status) find max(Q) for next_state-actions. Correct target by reward + gamma * max(Q).

*Make mini-batch learning passing the updated target to the network. 

*Make an estimate of an action, following e-greedy policy, for currently observed state; save the updated network.

*Make action in an emulator, observe reward (maybe zero == no reward), and next state. Save the new sequence to a random row in replay buffer when reward is non-zero (meaning, the sequence of state-actions riched its target).
Question_2: is it OK that my rewards can be negatively biased in the initial replay buffer?
Question_3: should I expect increasing max(Q) predicted over time as the network converges?
Question_4: I am not sure if I should use normalization (scaling) of max(Q) OR reward + max(Q) as an output for a relu-NN.
 A: Question 1
1) You initialize your network here and use it to dump the memory replay.
2) What is random sparse sample here? Your memory replay consists of N elements. You just sample randomly K of them, where K is the minibatch size.
3) correct, but the net initialization should happen on the step 1).
4) If you do training on the previous step, why do you want to do the update again?
5) okay
6) Several issues here. Why do you want to save a transition in a random place of the memory replay? You will sample it randomly anyway, why do you want to do this? 
You save the transition in the memory replay even if your reward is zero.  
Question 2
 What do you mean by rewards being negatively biased?
Question 3
Sure. You should expect that. What does you Q function give you? It gives you the future discounted reward flow, right? The better you play, the more reward you should get.
Question 4
Not sure, that I get it correctly, can you elaborate more on that?
I recommend implementing the algorithm for some simple gym environment, and then adjusting it for your specific case. If you have any questions, I will be glad to help.
