What can cause a cost function to look like this when using gradient descent?

Question

I am training a 3 layer neural network using back propagation (in python3). For that, I randomly initialize the weights in the network using numpy.random.randn(). I then use gradient descent to minimise the loss function. When I plot my cost function v/s the number of iterations, for different initial values of the weight matrix, I get different plots, looking like these. What could be a possible reason ? Can it be the case that there are two local minima in the vicinity, and based on the initial value of the weights, the descent step goes in different directions. Some times it's pretty good. But sometimes it's very nasty, something like this : The two labels being 0 and 1. The loss function is cross-entropy loss. Activation function is sigmoid on all nodes in the network.

Answer 1

Craziness like after iteration ~150 on your second chart tends to be due to bad metaparameters (specifically the learning rate, or momentum if you're using it). The descent is overshooting convergence because the learning (or momentum) is too high. Bad regularization can amplify this effect as well. Cross-validation is typically used to tune them.

I'm not sure what's going on before iteration ~150 there (referring specifically to the four curves). It looks like there are multiple loss values for single iterations. If that is the case, the only way that could get plotted is if multiple loss values are being calculated and stored for single iterations.