Assuming a simple, fully connected Multilayer Perceptron network with one input layer, one hidden layer with multiple nodes and one output layer. In this case the nodes in hidden layer are interchangable apart from the weight and bias values. Yet, based on the same inputs, each node will produce different outputs for the same input after training.
Similarily, in convolution networks different feature maps produce very different outputs (http://www.matthewzeiler.com/wp-content/uploads/2017/07/eccv2014.pdf), although they are connected to exactly the same input nodes in my understanding.
Are these different outputs purely determined due to the random weights during initialization and subsequent training?
(My intuition is rather reluctant to believe that, saying that training should rather make the output of the nodes more similar instead of more different. Maybe somebody has a nice example?)