0
$\begingroup$

I'm currently working on a Dueling-Double DQN model, and I noticed that though the loss (mse of Q values between training and target networks) seems to be decreasing, the distribution of weights in the network almost didn't change, while the biases varies a lot during training, resulting in Q values update.

basic setup:

hidden layers: 2

input nodes(state space): 80

hidden nodes: 256

output nodes(action space): 3

optimizer: adam

activation: relu

Since I'm using relu as activation function so I suppose gradient vanishing is not the reason, what other reason could cause the behaviour? Is there any direction for hyperparameter tuning since grid search costs a lot of time. Thanks.

$\endgroup$
1
$\begingroup$

Since I'm using relu as activation function so I suppose gradient vanishing is not the reason, what other reason could cause the behaviour?

Biases can directly move the output of your network no a non-zero mean level of Q value, when your problem is designed such that Q values are positively or negatively biased.

Moreover, I read a couple of opinions that adding bias weights is not a good option in neural network Q learning, since they introduce the bias in estimates, making NN outputs dependent. (This is not a solid recommendation though.)

The reason why other weigths do not change a lot is that learning your Q value is very noisy and any change in weights is pointless.

Is there any direction for hyperparameter tuning since grid search costs a lot of time.

Approaches exist. You can for example use a general purpose optimization method to tune hyperparameters to maximize the NN convergence (maximization of Q value). I believe some form of gradient descent can be an option with some limitations.

$\endgroup$
  • $\begingroup$ Thank you Alexey! My problem is indeed biased, negative rewards are common, while the positive rewards are quite rare. $\endgroup$ – Kevin. Fang Jul 13 '18 at 1:09
  • $\begingroup$ As I'm already using adam optimizer, which should have same level performance of SGD. So I assume I should tune the learning rate/batch size to get a better performance, but the weights will still not change much since changes in weights are still noisy? $\endgroup$ – Kevin. Fang Jul 13 '18 at 1:34
  • $\begingroup$ Try to run the best found found architecture, but turn off the bias terms in NN. In my experience hyperparameters do not influence much the results. $\endgroup$ – Alexey says Reinstate Monica Jul 13 '18 at 7:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.