0
$\begingroup$

I am currently reading Deepminds "Playing Atari with Deep Reinforcement Learning"-Paper (2013) and since I don't quite understand it, I would like to ask here.

What exactly happens in each fully connected layer? I understand what the convolutional layers do. How do the fully connected layers decide on which action to take?

From my understanding, we train, using the shown algorithm in the paper and when we finally got it trained, we just simply combine the information we get from the input with the weights and it's done.

Please help me :)

$\endgroup$
3
  • $\begingroup$ The whole neural network provides an estimate of the Q-function ( state action value function) from input images. Its last layer would give estimated utility of taking different actions. Based on the utilities, the agent decides to take the optimal action, or with a small probability take a random action (exploration). $\endgroup$
    – Lii
    Commented Dec 20, 2017 at 13:05
  • $\begingroup$ That I already know. I would like to know where the Q-function specifically is used in that fully connected layer only. I understand the basic conecpt if that wasn't clear. $\endgroup$
    – SirQ
    Commented Dec 20, 2017 at 13:10
  • $\begingroup$ I am not sure what you mean by "where the Q-function specifically is used in that fully connected layer". The fully connected layer does not "use" Q-function. But the loss function does involve Q-function. Perhaps that's what you are asking? $\endgroup$
    – Lii
    Commented Dec 22, 2017 at 16:05

1 Answer 1

1
$\begingroup$

The network in the referenced DQN paper estimates state-action values for a single state and all possible actions at once. It returns a vector $[\hat{q}(S,a_1), \hat{q}(S,a_2), \hat{q}(S,a_3)...]$ given an input state $S$ - the action space is the same for all states, and is just associated with each possible controller input (plus I assume a "no input" choice).

The action choice is then made using $\text{argmax}_a$ over this vector.

What exactly happens in each fully connected layer? I understand what the convolutional layers do. How do the fully connected layers decide on which action to take?

They don't "decide" what action to take, but calculate the action value for each $a$ - the fully connected layers are just a normal multi-layer neural network used for regression, taking the feature maps extracted from the pixels as input. The actual decision is left to the agent, but the predictions of the network guide the agent to the best estimate of the optimal action given any input state.

An alternative, and equally valid, approach is to have the network predict value given an (S, A) pair. Then in order to select an action, the network needs to predict a minibatch of the state plus all possible actions in order to construct the same vector, and select the maximum.

Depending on the nature of the environment and possible actions, you may choose either architecture (predict values of all actions at once from state representation or predict one action at at time from state & action representation), they are roughly equivalent theoretically.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.