I have implemented deep q network(DQN) from atari paper and running it on a different RL task. I'm plotting average q-value for 100k sample from 1kk dataset every once in awhile as suggested in the paper, because it should be more smooth than task score plot.
'*' and '--' is for test q-values and '-' and 'o' is for train q-values and thick lines are moving averages.
Blue network is deep but doesn't have too many neurons, pink network has a lot of neurons and layers. and green has small number of neurons and deep.
q-values do not reflect scores that networks are getting. Pink one has top score on training dataset out of these three. Score plot of all three networks All three networks get score that is below zero. And I need above zero.
I'm plotting it for train and test data sets and after awhile test plot starts to increase faster than train plot, is this overfitting? When I'm testing it on the test dataset, score is indeed lower than on train dataset.
Another thing, all my plots are positive(there was negative plot a few times but it quickly got to positive side) and grow, but when I test them - they give negative scores. Why my q-value plots are not negative?
Is my network underfitted and thinks that it's doing good when in reality it is not?
or not enough time training?
If I try to increase network depth or neurons number - I get crazy value plots like pink. And overfitting on the test dataset.