I am using Neural Network Q-value approximation in my reinforcement learning task. The approach is exactly the same as one described in this question, however the question itself is different.
In this approach the number of outputs is the number of actions we can take. And in simple words, the algorithm is following: do the action A, explore the reward, ask NN to predict Q values for all possible actions, choose maximum Q value, calculate Q for particular action A as
R + max(new_state_Q). Fit model on predicted Q values with only one of them replaced by
R + max(new_state_Q).
Question: How efficient is this approach if the number of outputs is big?
Attempt: Let's say there are 10 actions we can take. At each step we ask the model to predict 10 values, at early age of the model this prediction is total mess. Then we modify 1 value of the output and fit the model on these values.
I have two opposite thoughts on how good\bad is this approach and can't decide which one is right:
- From one point of view, we are training each neuron 9 times on a random data and only once on the data that is close to real value. If NN predicted 5 for action A in state S, but real value is -100 we will fit NN 9 times with value 5 and then once with value -100. Sounds crazy.
- From other point of view, the learning of neural network is implemented as back propagation of an error, so when model has predicted 5 and we are training it on 5 it will not learn anything new, as the error is 0. Weights are not touched. And only when we will calculate -100 and fit it to the model, it will do the weight recalculation.
Which option is right? Maybe there is something else I am not taking into account?
UPDATE: By "how efficient" I mean comparing to an approach with one output - predicted reward. Of course, the the action will be a part of input in this case. So approach #1 makes predictions for all actions based on some state, approach #2 makes prediction for specific action taken at some state.