What could be the causes of weights not changing during backpropagation?

Question

I'm currently working on a Dueling-Double DQN model, and I noticed that though the loss (mse of Q values between training and target networks) seems to be decreasing, the distribution of weights in the network almost didn't change, while the biases varies a lot during training, resulting in Q values update.

basic setup:

hidden layers: 2

input nodes(state space): 80

hidden nodes: 256

output nodes(action space): 3

optimizer: adam

activation: relu

Since I'm using relu as activation function so I suppose gradient vanishing is not the reason, what other reason could cause the behaviour? Is there any direction for hyperparameter tuning since grid search costs a lot of time. Thanks.

Answer 1

Since I'm using relu as activation function so I suppose gradient vanishing is not the reason, what other reason could cause the behaviour?

Biases can directly move the output of your network no a non-zero mean level of Q value, when your problem is designed such that Q values are positively or negatively biased.

Moreover, I read a couple of opinions that adding bias weights is not a good option in neural network Q learning, since they introduce the bias in estimates, making NN outputs dependent. (This is not a solid recommendation though.)

The reason why other weigths do not change a lot is that learning your Q value is very noisy and any change in weights is pointless.

Is there any direction for hyperparameter tuning since grid search costs a lot of time.

Approaches exist. You can for example use a general purpose optimization method to tune hyperparameters to maximize the NN convergence (maximization of Q value). I believe some form of gradient descent can be an option with some limitations.