General problem with correlated inputs: In general, it is possible to have correlated input variables, which leads to correlated weights. Let's take an extreme example and lets assume you have a duplicate feature, $x_1 = x_2$ (perfect correlation), and you want a linear function that maps $X$ to $Y$, $Y = f(X)$, where $f(X) = \beta_0 + \beta_1x_1 + \beta_2x_2$.
Assume $\beta_1 = 0, \beta_2 = 1$ gives an answer that is "ok"; but so does $\beta_1 = 1, \beta_2 = 0$, and every $\beta_1 + \beta_2 = 1$ will give the same answer.
This means that changing $\beta_1$ has a significant effect on the value $\beta_2$ should take. As Neural Networks compute such a linear model at each node, it can happen a lot from a low correlation to begin with.
Neural Net specifics: There is also another problem, more specific to the structure of neural networks, that will lead to this. Again, the extreme exemple: if two nodes in the same layer are initialized with the same values, they will have the same gradient at each iteration and learn the same transformed feature out of the last layer, leading to a duplicate.
Effect on gradient descent: This argument might be a bit loose, but Gradient Descent works 'best' when the direction of the gradient at each iteration points to the optimal point; that is, you could minimize each $\beta_i$ separately and get to a good answer. This is possible when the function to optimize is stricly convex. But when inputs are highly correlated, this is no longer the case. Obviously, it is not possible for neural networks since the function is not convex to begin with, but it has effects on reaching the local minimum as well.
A little bit off-topic, but to help figure out what is going on in gradient descent, you can check lecture 6.2 of Geoffrey Hinton's mooc; "A Bag of tricks for gradient descent" (Here on Coursera, Here on Youtube). He is not stricly talking about correlation, but he shows the effect of scaling and shifting the inputs, which helps get an intuition about how gradient descent works.