Visualizing model trajectories for Neural Networks using function approximator

Question

Erhan et al. in their 2010 paper discusses how pre-training improves deep networks: http://www.jmlr.org/papers/volume11/erhan10a/erhan10a.pdf#page=15

In there, they compare different neural network models by visualizing the function representation for each network. For a given model, Function representation is defined as a vector of outputs for a given set of inputs (the link above points directly to the description of function representation).

Inline below is a figure comparing model trajectories:

Question: Since, function representation is defined as a vector of output values for a given set of inputs, I assume there exists a vector $V_{c}$ with all the correct outputs. Now ideally, both the model trajectories (with pre-training and without pre-training) should try to converge towards this correct vector $V_{c}$ with progression in training iterations.

However, if we look at the figure above, the model trajectories seem to diverge, instead of converging towards a correct vector $V_{c}$. Why are the trajectories diverging instead of converging?

PS: Here is a link to a video that describes this figure: https://youtu.be/MJs9JHr8C-s?t=177

Answer 1

However, at some point (after about 7 epochs) the different trajectories (corresponding to different random seeds) diverge (slowing down into elongated jets) and never get back close to each other (this is more true for trajectories of networks without pre-training). This suggests that each trajectory moves into a different apparent local minimum.

So yes, there exists some correct vector $V_c$, however, both paths encounter completely different local minima and they get stuck there.