This is the issue that I am still unclear about in Neural Networks, and I was wondering how much of an open issue it still in in Deep Learning research. How can neural networks learn when the nodes in each layer are essentially identical? I found some similar questions, but I was hoping for a more precise and mathematical answer.
QUESTION SETUP
Say I have a very simple fully connected network with two hidden layers, and an input and output layer, such as in the diagram below, taken from this post.
Now, all of the nodes in the first hidden layer (blue) have the same exact equation, meaning that they will have the same function $z = W_{ij}*X + bias_i$ where $i \in {1...I}$ layers and $j \in {1...J}$ node in that layer. Each of these nodes also has the same non-linear activation function, such as RELU.
So the nodes in a given layer are essentially identical in structure because the layer is fully connected. The only difference is that the weights are initialized at different values, and hence it takes time for those weights to adapt through backpropagation.
QUESTION
So each node in a given layer has the same exact equation, but varies only in initialization weights and the trajectory that the optimizer takes on that node. So over time, should not all of the weights for each node arrive at the same exact set of weights? If each node is getting the same input values, then why do these nodes not all converge to the same weight matrix over time? Seem like this is the key point of neural networks, that the nodes will specialize to focus on different parts of the task. But I was not sure how this works in practice when all of the nodes seem identical?
Even in CNNs and RNNs they often have fully connected layers at the top. So how is this divergence happening when the node equations are identical? I mean if I ran a bunch of multiple linear regressions on the same $X$ values, I would expect to get the same answer for each run of that regression equation.
Really I am looking for any research papers or such that anyone can identify on this topic. I would really like to see the mathematical proofs behind this idea. I think that there is a lot going on here, but I would really like to understand this critical topic in training neural nets.