0
$\begingroup$

I was tweaking the Tensorflow MNIST Tutorial, and I am not clear why the weights are initialized using tf.zeros. If I switch to using tf.random_normal, or tf.truncated_normal, I see a very significant drop in accuracy.

Any intuition why tf.zeros is better initial value of the weight and biases (even tf.ones perform the same as tf.zeros), than random values?

Conversely, it seems that if I want to extend the network to have a hidden layer, the weights of the hidden layer should be intialized with random values instead of zeros (unlike the first layer) to do well.

Eg.

x = tf.placeholder(tf.float32, [None, 784])
W = tf.Variable(tf.zeros([784, 100]))
b = tf.Variable(tf.zeros([100]))
h1 = tf.sigmoid(tf.matmul(x, W) + b)

W2 = tf.Variable(tf.random_normal([100, 10]))
b2 = tf.Variable(tf.zeros([10]))
y = tf.matmul(h1, W2) + b2

W2 should be intiailized using tf.random_normal (accuracy > 92%) instead of tf.zeros (accuracy ~ 25%).

Edit: My question is not a duplicate, because unlike in the tagged question, I understand why initializing with random weights will help. It is counterintuitive why initializing with zeros does much better for the first layer, but not for the subsequent layers?

Improve this question
$\endgroup$
4
  • $\begingroup$ Replied in the edits. I am more interested in understanding why tf.zeros is a better choice than random initialization of the weights for the first layer, but not for any subsequent layer. $\endgroup$
    – reddragon
    Dec 3, 2017 at 17:43
  • $\begingroup$ Did you actually read the answers? In order for Gaussian initialization to work, you can't just set the standard deviation arbitrarily. Which standard deviation are you using here? $\endgroup$
    – DeltaIV
    Dec 3, 2017 at 18:45
  • $\begingroup$ Thanks for the comment DeltaIV. I am using the default stddev of 1.0 for random_normal (tensorflow.org/api_docs/python/tf/random_normal). However, correct me if I am wrong, but tf.random_normal, should work better than tf.zeros, since initializing with all zero weights would lead to all the first layer outputs being zeros. However, the network seems to learn quite well, which is confounding. $\endgroup$
    – reddragon
    Dec 4, 2017 at 17:33
  • $\begingroup$ I think I figured out why does it work with tf.zeros. It is because the activation function is sigmoid, which has a non-zero output (0.5) for a zero input. So all the first layer inputs will have the same non-zero output (which is not great, but it works). It will indeed fail if the activation was ReLU. $\endgroup$
    – reddragon
    Dec 4, 2017 at 18:43

0

Reset to default

Browse other questions tagged or ask your own question.