What is the difference between calculations and output of individual neurons in a layer of neural network? Consider a simplest possible neural network with two layers, Input with 2 features, a hidden layer with 2 neurons with sigmoid activation function and output layer with single neuron with activation function sigmoid. Me being newbie, trying to understand in hidden layer, what makes the calculations in neuron 1 different than neuron 2? As I understand initial weights are randomly assigned along with the input features itself and same will be fed to each neurons. Even bias also the same. So can we assume in first level of calculation all outputs are same from both neurons and changes start to happen only after back propagation?
 A: Different combinations can give the same output value!
Consider a simple neural network with one feature, a hidden layer with two neurons (ReLU activation), and an output neuron with an identity activation.
$$
\widehat{y_i} = \widehat{b_{1,2}} + \widehat{w_{1,2}}ReLU\bigg(
\widehat{b_{1,1}}+\widehat{w_{1,1}}x
\bigg)
+ \widehat{w_{2,2}}ReLU\bigg(
\widehat{b_{2,1}}+\widehat{w_{2,1}}x
\bigg)
$$
The following two sets of parameter estimates give the same output values of $\widehat{y_i}$.
1
$
\widehat{b_{1,2}} = 1\\
\widehat{w_{1,2}} = 2\\
\widehat{b_{1,1}} = 3\\
\widehat{w_{1,1}} = 4\\
\widehat{w_{2,2}} = 5\\
\widehat{b_{2,1}} = 6\\
\widehat{w_{2,1}} = 7
$
2
$
\widehat{b_{1,2}} = 1\\
\widehat{w_{1,2}} = 5\\
\widehat{b_{1,1}} = 6\\
\widehat{w_{1,1}} = 7\\
\widehat{w_{2,2}} = 2\\
\widehat{b_{2,1}} = 3\\
\widehat{w_{2,1}} = 4$
That is, neural network solutions in general are not unique.
A: The weight of the neuron and the bias term are what differentiate the neurons. Since you start randomly, unless both neurons are assigned the same weight and bias it would not guarantee the same outputs. 
If they are assigned the same weight and bias there might be something wrong with your randomization.
