Consider a simplest possible neural network with two layers, Input with 2 features, a hidden layer with 2 neurons with sigmoid activation function and output layer with single neuron with activation function sigmoid. Me being newbie, trying to understand in hidden layer, what makes the calculations in neuron 1 different than neuron 2? As I understand initial weights are randomly assigned along with the input features itself and same will be fed to each neurons. Even bias also the same. So can we assume in first level of calculation all outputs are same from both neurons and changes start to happen only after back propagation?
The weight of the neuron and the bias term are what differentiate the neurons. Since you start randomly, unless both neurons are assigned the same weight and bias it would not guarantee the same outputs.
If they are assigned the same weight and bias there might be something wrong with your randomization.