How efficient is Q-learning with Neural Networks when there is one output unit per action?

Question

Background:
I am using Neural Network Q-value approximation in my reinforcement learning task. The approach is exactly the same as one described in this question, however the question itself is different.

In this approach the number of outputs is the number of actions we can take. And in simple words, the algorithm is following: do the action A, explore the reward, ask NN to predict Q values for all possible actions, choose maximum Q value, calculate Q for particular action A as R + max(new_state_Q). Fit model on predicted Q values with only one of them replaced by R + max(new_state_Q).

Question: How efficient is this approach if the number of outputs is big?

Attempt: Let's say there are 10 actions we can take. At each step we ask the model to predict 10 values, at early age of the model this prediction is total mess. Then we modify 1 value of the output and fit the model on these values.

I have two opposite thoughts on how good\bad is this approach and can't decide which one is right:

• From one point of view, we are training each neuron 9 times on a random data and only once on the data that is close to real value. If NN predicted 5 for action A in state S, but real value is -100 we will fit NN 9 times with value 5 and then once with value -100. Sounds crazy.
• From other point of view, the learning of neural network is implemented as back propagation of an error, so when model has predicted 5 and we are training it on 5 it will not learn anything new, as the error is 0. Weights are not touched. And only when we will calculate -100 and fit it to the model, it will do the weight recalculation.

Which option is right? Maybe there is something else I am not taking into account?

UPDATE: By "how efficient" I mean comparing to an approach with one output - predicted reward. Of course, the the action will be a part of input in this case. So approach #1 makes predictions for all actions based on some state, approach #2 makes prediction for specific action taken at some state.

Answer 1

So the two options we want to compare are:

1. Inputs = state representation, Outputs = 1 node per action
2. Inputs = state representation + one-hot encoding of actions, Outputs = 1 node

Going by my own intuition, I doubt there's a significant difference in terms of representation power or learning speed (in terms of iterations) between those two options.

For representation power, the first option gives a slightly ''smaller'' network near the inputs, and a ''wider'' network near the outputs. If for whatever reason it were beneficial to have more weights close to the input nodes for example, that could pretty much be achieved by making the first hidden layer (close to the inputs) a bit bigger too.

As for learning speed, the concern you seem to have is basically along the lines of generally only having an accurate learning signal for one of the outputs, and not for the others. With the second option, exactly the same can be said for weights connected to input nodes though, so I doubt there's a significant difference there.

Like I mentioned, all of the above is based just on my intuition though, would be interesting to see more credible references on that.

One important advantage I see for the first option is in computational speed; suppose you want to compute $Q$-values for all actions in order to decide which action to select; a single forwards pass through the network, giving you all the $Q$-values at once, will be much more efficient computationally than having $n$ separate forwards passes (for an action set of size $n$).