starting off with same weights in neural networks

Question

In a simple neural network with two neurons in series carrying weights w1 and w2 and just one input, what is limitation behind starting off with both weights being same?

I am referring to patrick winston's MIT lectures and to quote him ".. By the way what would happen if you started off with all the weights being the same? Nothing, because it would always stay the same"

Since the two neurons are in series i fail to see why they would necessarily remain same

Thank you in advance

Answer 1

Firstly, see the answer to this post

Furthermore, to expand on that answer, if all hidden neurons and subsequently all output neurons have the same activation values, there would only be $k$ different updates for the weights between the hidden and the output layer where $k$ is the number of output neurons. This is easily understandable by looking at the update rule.

This is bad because the power of neural networks comes from their ability to associate different weights to different neurons (parameters) and by initializing the network with equal weights, you are forcing it to yield groups of same-valued weights.

Answer 2

You're right for two neurons in series. The derivatives would be different after the first step, at least for sigmoid activation.

EDIT: I've done a quick check using Tensorflow, you can see below that it learns the autoencoder fine. I noticed that if you use batch gradient descent, and have exactly the same number of positive and negative instances, then it can't learn anything. The batch gradient is all-zero (a critical point) in that case. That might be the setting he is referring to.

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import math

import tensorflow as tf

sess = tf.Session()

somedata = [1,0,1,0,1,0,0,1,1,0,0,0]
somedata_col = [[x] for x in somedata]

#input_data = tf.constant(somedata)
input_value = tf.placeholder(tf.float32, [None, 1])
output_value = tf.placeholder(tf.float32, [None, 1])

b1 = tf.Variable(0.0, name="b1")
w1 = tf.Variable([[0.0]], name="w1")
a1 = b1 + tf.matmul(input_value, w1)
z1 = tf.sigmoid(a1, name="z1")

b2 = tf.Variable(0.0, name="b2")
w2 = tf.Variable([[0.0]], name="w2")
a2 = b2 + tf.matmul(z1, w2) #tf.identity(, name="a2")
z2 = tf.sigmoid(a2, name="z2")

#cost = tf.square(a2 - output_value, name="cost")
cost = tf.reduce_sum(-output_value*tf.log(z2) - (1-output_value)*tf.log(1-z2))

fd = { input_value: somedata_col, output_value: somedata_col}

optimizer = tf.train.GradientDescentOptimizer(0.5)
train = optimizer.minimize(cost, var_list=[b1,w1,b2,w2])

init = tf.global_variables_initializer()
sess.run(init)
for step in xrange(20):
    print("step", step, "weights:", sess.run([b1,w1,b2,w2]))
    sess.run(train, feed_dict=fd)

print("predictions:", z2.eval(fd, session=sess))

Here is the output:

step 0 weights: [0.0, array([[ 0.]], dtype=float32), 0.0, array([[ 0.]], dtype=float32)]
step 1 weights: [0.0, array([[ 0.]], dtype=float32), -0.5, array([[-0.25]], dtype=float32)]
step 2 weights: [-0.025508083, array([[-0.1017742]], dtype=float32), -0.091870666, array([[-0.04593527]], dtype=float32)]
step 3 weights: [-0.021670617, array([[-0.1168806]], dtype=float32), -0.42096692, array([[-0.24193409]], dtype=float32)]
step 4 weights: [-0.038621157, array([[-0.21173379]], dtype=float32), -0.13332921, array([[-0.14563146]], dtype=float32)]
step 5 weights: [-0.030633193, array([[-0.26093858]], dtype=float32), -0.3320294, array([[-0.31528473]], dtype=float32)]
step 6 weights: [-0.044254888, array([[-0.37951675]], dtype=float32), -0.12732345, array([[-0.31395262]], dtype=float32)]
step 7 weights: [-0.031692233, array([[-0.4850899]], dtype=float32), -0.22682101, array([[-0.49384922]], dtype=float32)]
step 8 weights: [-0.03928249, array([[-0.65885508]], dtype=float32), -0.069601834, array([[-0.59461039]], dtype=float32)]
step 9 weights: [-0.012473229, array([[-0.84564459]], dtype=float32), -0.091074526, array([[-0.82879764]], dtype=float32)]
step 10 weights: [0.0079552941, array([[-1.09850502]], dtype=float32), 0.049069822, array([[-1.04975295]], dtype=float32)]
step 11 weights: [0.086042896, array([[-1.37198019]], dtype=float32), 0.093280971, array([[-1.374174]], dtype=float32)]
step 12 weights: [0.18385497, array([[-1.69305539]], dtype=float32), 0.2489447, array([[-1.71299231]], dtype=float32)]
step 13 weights: [0.35441414, array([[-2.02026224]], dtype=float32), 0.36500531, array([[-2.11956358]], dtype=float32)]
step 14 weights: [0.53190947, array([[-2.3695395]], dtype=float32), 0.57182026, array([[-2.52639461]], dtype=float32)]
step 15 weights: [0.74612176, array([[-2.70178413]], dtype=float32), 0.75401825, array([[-2.95842862]], dtype=float32)]
step 16 weights: [0.92347378, array([[-3.02649689]], dtype=float32), 0.98713511, array([[-3.36072731]], dtype=float32)]
step 17 weights: [1.1052706, array([[-3.31118464]], dtype=float32), 1.1785023, array([[-3.75399303]], dtype=float32)]
step 18 weights: [1.2446547, array([[-3.57015085]], dtype=float32), 1.3825138, array([[-4.10399199]], dtype=float32)]
step 19 weights: [1.3762732, array([[-3.79116011]], dtype=float32), 1.5515869, array([[-4.42910194]], dtype=float32)]
predictions: [[ 0.7950502 ]
 [ 0.10589811]
 [ 0.7950502 ]
 [ 0.10589811]
 [ 0.7950502 ]
 [ 0.10589811]
 [ 0.10589811]
 [ 0.7950502 ]
 [ 0.7950502 ]
 [ 0.10589811]
 [ 0.10589811]
 [ 0.10589811]]