In a simple neural network with two neurons in series carrying weights w1 and w2 and just one input, what is limitation behind starting off with both weights being same?

I am referring to patrick winston's MIT lectures and to quote him ".. By the way what would happen if you started off with all the weights being the same? Nothing, because it would always stay the same"

Since the two neurons are in series i fail to see why they would necessarily remain same

Thank you in advance


Firstly, see the answer to this post

Furthermore, to expand on that answer, if all hidden neurons and subsequently all output neurons have the same activation values, there would only be $k$ different updates for the weights between the hidden and the output layer where $k$ is the number of output neurons. This is easily understandable by looking at the update rule.

This is bad because the power of neural networks comes from their ability to associate different weights to different neurons (parameters) and by initializing the network with equal weights, you are forcing it to yield groups of same-valued weights.

  • $\begingroup$ @NitinSiwach Please check the link you provided, it is not relevant to the question $\endgroup$ – Alex P Apr 3 '17 at 15:26
  • $\begingroup$ im sorry. commenting again: dont think this answers why, when two neurons are in series, is it that same weight would be a problem. youtube.com/watch?v=uXt8qF2Zzfo 28:14 would show you the NNet in action. 41:01-41:07 would have the prof make the statement about this particular NNet. am i missing something? $\endgroup$ – Nitin Siwach Apr 4 '17 at 0:20
  • $\begingroup$ In his network, it is not possible to see what effect this specific initialization has. To be more exact, there is no effect in how the network learns, for this particular network. You might be able to train that network faster by initializing with a different set of weights given a problem, but that has to do more with chance (let's not go into learning slowdown cases). However, in the general case, my original answer stands and this initialization case will, in general (well, in almost all real-world applications), cause very poor network performance. $\endgroup$ – Alex P Apr 4 '17 at 15:58

You're right for two neurons in series. The derivatives would be different after the first step, at least for sigmoid activation.

EDIT: I've done a quick check using Tensorflow, you can see below that it learns the autoencoder fine. I noticed that if you use batch gradient descent, and have exactly the same number of positive and negative instances, then it can't learn anything. The batch gradient is all-zero (a critical point) in that case. That might be the setting he is referring to.

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import math

import tensorflow as tf

sess = tf.Session()

somedata = [1,0,1,0,1,0,0,1,1,0,0,0]
somedata_col = [[x] for x in somedata]

#input_data = tf.constant(somedata)
input_value = tf.placeholder(tf.float32, [None, 1])
output_value = tf.placeholder(tf.float32, [None, 1])

b1 = tf.Variable(0.0, name="b1")
w1 = tf.Variable([[0.0]], name="w1")
a1 = b1 + tf.matmul(input_value, w1)
z1 = tf.sigmoid(a1, name="z1")

b2 = tf.Variable(0.0, name="b2")
w2 = tf.Variable([[0.0]], name="w2")
a2 = b2 + tf.matmul(z1, w2) #tf.identity(, name="a2")
z2 = tf.sigmoid(a2, name="z2")

#cost = tf.square(a2 - output_value, name="cost")
cost = tf.reduce_sum(-output_value*tf.log(z2) - (1-output_value)*tf.log(1-z2))

fd = { input_value: somedata_col, output_value: somedata_col}

optimizer = tf.train.GradientDescentOptimizer(0.5)
train = optimizer.minimize(cost, var_list=[b1,w1,b2,w2])

init = tf.global_variables_initializer()
for step in xrange(20):
    print("step", step, "weights:", sess.run([b1,w1,b2,w2]))
    sess.run(train, feed_dict=fd)

print("predictions:", z2.eval(fd, session=sess))

Here is the output:

step 0 weights: [0.0, array([[ 0.]], dtype=float32), 0.0, array([[ 0.]], dtype=float32)]
step 1 weights: [0.0, array([[ 0.]], dtype=float32), -0.5, array([[-0.25]], dtype=float32)]
step 2 weights: [-0.025508083, array([[-0.1017742]], dtype=float32), -0.091870666, array([[-0.04593527]], dtype=float32)]
step 3 weights: [-0.021670617, array([[-0.1168806]], dtype=float32), -0.42096692, array([[-0.24193409]], dtype=float32)]
step 4 weights: [-0.038621157, array([[-0.21173379]], dtype=float32), -0.13332921, array([[-0.14563146]], dtype=float32)]
step 5 weights: [-0.030633193, array([[-0.26093858]], dtype=float32), -0.3320294, array([[-0.31528473]], dtype=float32)]
step 6 weights: [-0.044254888, array([[-0.37951675]], dtype=float32), -0.12732345, array([[-0.31395262]], dtype=float32)]
step 7 weights: [-0.031692233, array([[-0.4850899]], dtype=float32), -0.22682101, array([[-0.49384922]], dtype=float32)]
step 8 weights: [-0.03928249, array([[-0.65885508]], dtype=float32), -0.069601834, array([[-0.59461039]], dtype=float32)]
step 9 weights: [-0.012473229, array([[-0.84564459]], dtype=float32), -0.091074526, array([[-0.82879764]], dtype=float32)]
step 10 weights: [0.0079552941, array([[-1.09850502]], dtype=float32), 0.049069822, array([[-1.04975295]], dtype=float32)]
step 11 weights: [0.086042896, array([[-1.37198019]], dtype=float32), 0.093280971, array([[-1.374174]], dtype=float32)]
step 12 weights: [0.18385497, array([[-1.69305539]], dtype=float32), 0.2489447, array([[-1.71299231]], dtype=float32)]
step 13 weights: [0.35441414, array([[-2.02026224]], dtype=float32), 0.36500531, array([[-2.11956358]], dtype=float32)]
step 14 weights: [0.53190947, array([[-2.3695395]], dtype=float32), 0.57182026, array([[-2.52639461]], dtype=float32)]
step 15 weights: [0.74612176, array([[-2.70178413]], dtype=float32), 0.75401825, array([[-2.95842862]], dtype=float32)]
step 16 weights: [0.92347378, array([[-3.02649689]], dtype=float32), 0.98713511, array([[-3.36072731]], dtype=float32)]
step 17 weights: [1.1052706, array([[-3.31118464]], dtype=float32), 1.1785023, array([[-3.75399303]], dtype=float32)]
step 18 weights: [1.2446547, array([[-3.57015085]], dtype=float32), 1.3825138, array([[-4.10399199]], dtype=float32)]
step 19 weights: [1.3762732, array([[-3.79116011]], dtype=float32), 1.5515869, array([[-4.42910194]], dtype=float32)]
predictions: [[ 0.7950502 ]
 [ 0.10589811]
 [ 0.7950502 ]
 [ 0.10589811]
 [ 0.7950502 ]
 [ 0.10589811]
 [ 0.10589811]
 [ 0.7950502 ]
 [ 0.7950502 ]
 [ 0.10589811]
 [ 0.10589811]
 [ 0.10589811]]
  • $\begingroup$ No, he is talking about the neurons in series. Pretty sure of that. this is where he does that. youtube.com/watch?v=uXt8qF2Zzfo&t=949s at 41:00 (to get that he is talking about neurons in series, watch from 39:00) $\endgroup$ – Nitin Siwach Apr 3 '17 at 5:13
  • $\begingroup$ @Nitin I've updated my answer with code. $\endgroup$ – AaronDefazio Apr 4 '17 at 1:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.