I was reading through some blogs about Deep Q-Learning (DQN), and I have 2 questions:
1- I didn't understand how breaking the correlation between consecutive samples (i.e. train the network with random samples from Replay Memory, rather than providing it with the sequential experiences as they occur) would be a better learning process for the network!! Everywhere explains this in a very high level vague explanation. What I need to understand is how this affect the Backpropogation, Gradient decent, Weights or Loss calculation so that it makes learning inefficient. Perhaps mathematically or in a more evident way. I couldn't find a source that explains how and in what way the correlation of consecutive samples would decrease learning efficiency.
2- In one of the DQN blogs, in the
Wrapping up section, it explains the list of high level steps for the DQN process. What I don't understand is, after taking the first action from the initial state, I didn’t understand why we sample random batch from Replay Memory! Shouldn’t we continue from the next state where the agent ended up after taking the first action, as it is the natural progression in the game? I know Markov Property dictates that given the current state you can "ignore" all the past states and still be able to learn. But it doesn't make sense in an action/adventure/strategy real time game where the gameplay is continuous and sometimes require long term planning to end up in a better position. Say if it’s Super Mario game where the first action is a move forward. This will bring Mario closer to the first enemy Mashroom (Goomba) approaching. Shouldn’t we continue from the new state, by jumping over the enemy and then continue from there on? It doesn't make sense to suddenly sample some completely random batch state from the Replay Memory, which would make the agent end up in another part of the game, when the agent should continue progressing through the game as a human player would. I hope my question makes sense.
To mention another example, the DeepMind Alphastar AI agent that played against human players in the StarCraft 2 RTS game. This agent required to continue playing from the previous sequences of states to take new actions. So I don't understand where the random sampling of completely different states from the Replay Memory would come into play here. Maybe I'm missing something.
Many thanks in advance for any clarifications.