I am dealing with a process arising in time series. As an inspiration I refer to a paper by Mnih et al. Playing Atari with Deep Reinforcement Learning: https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf
Question 1: Am I following a general logic correctly?
Accumulate a replay buffer size N with sequences of state-action-reward-next_state* agent's behaviour from let's say ealier part of my time series.
- I use several successive sequences s-a-r until I hit a non-zero reward as one memory example, and I hardcoded the length of the sequences to allow easy passage to a neural network.
Select random sparse sample of size M from the buffer to teach the network. Split M into minibatches.
Initialize a new network if it did not exists, or use the network from previous time step (it's weight status) find max(Q) for next_state-actions. Correct target by reward + gamma * max(Q).
Make mini-batch learning passing the updated target to the network.
Make an estimate of an action, following e-greedy policy, for currently observed state; save the updated network.
Make action in an emulator, observe reward (maybe zero == no reward), and next state. Save the new sequence to a random row in replay buffer when reward is non-zero (meaning, the sequence of state-actions riched its target).
Question_2: is it OK that my rewards can be negatively biased in the initial replay buffer?
Question_3: should I expect increasing max(Q) predicted over time as the network converges?
Question_4: I am not sure if I should use normalization (scaling) of max(Q) OR reward + max(Q) as an output for a relu-NN.