I'm trying to diagnose overfitting in my multi-layer perceptron by looking at the weights, biases and gradients in each layer. I'm noticing that in the neural network that is overfitting, the weights in one of the hidden layers all have the same sign, and the majority of the biases in all the hidden layers have the same sign. In the network that isn't overfitting, the biases are centred on zero, with rougly equal numbers of positive and negative values. I'm using xavier initialization for both.
I think that the same signs of the biases or weights aren't the cause of overfitting in your network. Overfitting in a network occurs when the neurons become dependent to one another when learning. You need to make sure that your neurons are learning independently. Thus, you should try using dropouts. Dropouts remove or disable a certain percentage of neurons randomly.