# Why do nodes in hidden layer produce different results?

Assuming a simple, fully connected Multilayer Perceptron network with one input layer, one hidden layer with multiple nodes and one output layer. In this case the nodes in hidden layer are interchangable apart from the weight and bias values. Yet, based on the same inputs, each node will produce different outputs for the same input after training.

Similarily, in convolution networks different feature maps produce very different outputs (http://www.matthewzeiler.com/wp-content/uploads/2017/07/eccv2014.pdf), although they are connected to exactly the same input nodes in my understanding.

Are these different outputs purely determined due to the random weights during initialization and subsequent training?

(My intuition is rather reluctant to believe that, saying that training should rather make the output of the nodes more similar instead of more different. Maybe somebody has a nice example?)

* I suppose that specific regularization schemes could enforce "similar" weights, but that seems out of sync with the premise of the question. For example, strong $L^1$ penalties could cause a number of weights to be pinned very near to 0, but I don't think OP is asking about these kinds of corner cases.