Why do nodes in hidden layer produce different results?

Question

Assuming a simple, fully connected Multilayer Perceptron network with one input layer, one hidden layer with multiple nodes and one output layer. In this case the nodes in hidden layer are interchangable apart from the weight and bias values. Yet, based on the same inputs, each node will produce different outputs for the same input after training.

Similarily, in convolution networks different feature maps produce very different outputs (http://www.matthewzeiler.com/wp-content/uploads/2017/07/eccv2014.pdf), although they are connected to exactly the same input nodes in my understanding.

Are these different outputs purely determined due to the random weights during initialization and subsequent training?

(My intuition is rather reluctant to believe that, saying that training should rather make the output of the nodes more similar instead of more different. Maybe somebody has a nice example?)

Answer 1

Generically, when the network learns different weights for each node, it does so because the fit is better.* The optimization procedure has the goal of reducing the error, so if configurations with different weights are better, that's what the optimizer goes with.

But there's no particular reason that the weights must be the same or different. Consider the neural network with no hidden later and linear activations. This network is a regression problem, and it is convex provided that the design matrix is full rank. It could happen that, because of intentional design or happy accident, that two regression weights are statistically equal. I don't think there's anything especially remarkable about this in the regression context, nor would I be astonished if it happened in a neural network context either.

The one sticking point with this analysis is that, in general, neural networks are not convex, so local optimizers like gradient descent are not guaranteed to find the best solution. If the optimizer can't find the "best" solution, then we can't be certain that an alternative solution with different weights would have a lower loss.

---

* I suppose that specific regularization schemes could enforce "similar" weights, but that seems out of sync with the premise of the question. For example, strong $L^1$ penalties could cause a number of weights to be pinned very near to 0, but I don't think OP is asking about these kinds of corner cases.