For discrete action space, I thought the "conventional method" is to set up the neural network in such a way that the inputs are the states and each of the output node represents possible action with its value being the Q-value for that action, hence the optimal action is the action with the highest Q-value, that is, output node with the biggest value.
But for deep deterministic policy gradient, how to set up the Q function network in such a way that it is a function of the current state (or next state...etc) AND is also a function of the continuous output of the policy network? (like in the eqn 12 , 14?)
Judging from the algorithm, I would need additional input node that takes in the continuous output from the policy network and 1 output node for the Q value for its given state and action. Is that a correct interpretation?
Also, what about "multi-dimensional" action? So instead of the policy network having 1 output node with its continuous output, but rather a multiple output nodes ( which is useful for a robotic arm's angular velocity in X-axis, Y-axis.....etc)
Could I do the same but just extend the dimensions of the neural net?
Has anyone had experience with DDPG? Since I would need to store the replay buffer and have 2 additional network for target policy and target Q, I'd assume it would require a lot of memory. How is the overall training speed as well?