how is deep learning integrated into the reinforcement learning

Question

How is deep learning integrated into the reinforcement learning framework in deep Q learning?

Deep Q learning takes the Q learning, reinforcement learning ideas and mergers them with representation learning of deep neural network, could any one provide some details?

Answer 1

The core of Q-learning is to learn a function $Q$ which maps state-action pairs to the expected discounted future reward.

This function can be represented in a variety of ways

1. As a look-up table (each row containing state, action, and expected reward)
2. As a linear or nonlinear regression model mapping state-action to reward.

Deep Q-learning is simply an extension of representation 2 to using a much deeper and higher capacity regression model.

Answer 2

The input into the neural network is the current observation of the environment (for example, a screen shot of a game, or a list of values from some sensors).

The output from the neural network is a list of Q-values covering each of the choices that the agent can make (in Space Invaders for example, the list might be a vector of length 4, corresponding to "move left", "move right", "stop", "shoot").

So... how do you train the agent, starting with a randomised set of network weights? The algorithm goes something like this:

1. Input the environment observation into the network,
2. Store the observation and output Q-values in the GAME_MEMORY,
3. Make the agent act upon the highest Q-value, or take a random step depending on the value of a slowly degrading variable (epsilon),
4. Has game termination state been reached? If yes, get the game score and goto #5, otherwise goto #1.
5. Working backwards through the GAME_MEMORY (stepping from the last move to the first move) assign the game score value to the Q-value that was acted upon for each step, remembering to reduce the score value by some factor (discount_rate) for each step back through the history.
6. Add the GAME_MEMORY data to GAME_TRAINING_DATA. Clear GAME_MEMORY.
7. Have you played sufficient games to explore the state space properly? If yes, goto #8, otherwise goto #1
8. Using the environment observations recorded in GAME_TRAINING_DATA as input and the now updated Q-values as output, train your network. For example, if you played 75 games and each one took 10 steps to reach a termination condition, you now have 750 pieces of data on which to train your network.**
9. Clear the GAME_TRAINING_DATA.
10. Do you think that your agent is as good as it is going to get? If so, goto #11, otherwise goto #1.
11. Your network is now trained and should allow an agent to input its observations and receive back a list of Q-values, the highest valued of which corresponds to the optimum decision for what it should do.*

Here is a minimal implementation of this algorithm. It needs Python 3.x, TensorFlow and Keras to run it.

*Naturally, this depends if you have allowed your agent to play enough games that he has properly explored the problem space, and that your network is designed so that it can properly learn and generalise your observations...

**Some training regimes recommend using only a subset of the captured data for training, or randomising its order.