# What happens in the fully connected layer of a Deep-Q-Network?

I am currently reading Deepminds "Playing Atari with Deep Reinforcement Learning"-Paper (2013) and since I don't quite understand it, I would like to ask here.

What exactly happens in each fully connected layer? I understand what the convolutional layers do. How do the fully connected layers decide on which action to take?

From my understanding, we train, using the shown algorithm in the paper and when we finally got it trained, we just simply combine the information we get from the input with the weights and it's done.

• The whole neural network provides an estimate of the Q-function ( state action value function) from input images. Its last layer would give estimated utility of taking different actions. Based on the utilities, the agent decides to take the optimal action, or with a small probability take a random action (exploration).
– Lii
Dec 20, 2017 at 13:05
• That I already know. I would like to know where the Q-function specifically is used in that fully connected layer only. I understand the basic conecpt if that wasn't clear.
– SirQ
Dec 20, 2017 at 13:10
• I am not sure what you mean by "where the Q-function specifically is used in that fully connected layer". The fully connected layer does not "use" Q-function. But the loss function does involve Q-function. Perhaps that's what you are asking?
– Lii
Dec 22, 2017 at 16:05

The network in the referenced DQN paper estimates state-action values for a single state and all possible actions at once. It returns a vector $[\hat{q}(S,a_1), \hat{q}(S,a_2), \hat{q}(S,a_3)...]$ given an input state $S$ - the action space is the same for all states, and is just associated with each possible controller input (plus I assume a "no input" choice).
The action choice is then made using $\text{argmax}_a$ over this vector.
They don't "decide" what action to take, but calculate the action value for each $a$ - the fully connected layers are just a normal multi-layer neural network used for regression, taking the feature maps extracted from the pixels as input. The actual decision is left to the agent, but the predictions of the network guide the agent to the best estimate of the optimal action given any input state.