I was tweaking the Tensorflow MNIST Tutorial, and I am not clear why the weights are initialized using tf.zeros. If I switch to using tf.random_normal, or tf.truncated_normal, I see a very significant drop in accuracy.
Any intuition why tf.zeros is better initial value of the weight and biases (even tf.ones perform the same as tf.zeros), than random values?
Conversely, it seems that if I want to extend the network to have a hidden layer, the weights of the hidden layer should be intialized with random values instead of zeros (unlike the first layer) to do well.
Eg.
x = tf.placeholder(tf.float32, [None, 784])
W = tf.Variable(tf.zeros([784, 100]))
b = tf.Variable(tf.zeros([100]))
h1 = tf.sigmoid(tf.matmul(x, W) + b)
W2 = tf.Variable(tf.random_normal([100, 10]))
b2 = tf.Variable(tf.zeros([10]))
y = tf.matmul(h1, W2) + b2
W2
should be intiailized using tf.random_normal (accuracy > 92%) instead of tf.zeros (accuracy ~ 25%).
Edit: My question is not a duplicate, because unlike in the tagged question, I understand why initializing with random weights will help. It is counterintuitive why initializing with zeros does much better for the first layer, but not for the subsequent layers?