# What can cause a cost function to look like this when using gradient descent?

I am training a 3 layer neural network using back propagation (in python3). For that, I randomly initialize the weights in the network using numpy.random.randn(). I then use gradient descent to minimise the loss function. When I plot my cost function v/s the number of iterations, for different initial values of the weight matrix, I get different plots, looking like these. What could be a possible reason ? Can it be the case that there are two local minima in the vicinity, and based on the initial value of the weights, the descent step goes in different directions. Some times it's pretty good. But sometimes it's very nasty, something like this : The two labels being 0 and 1. The loss function is cross-entropy loss. Activation function is sigmoid on all nodes in the network.

• Did you write the network code yourself? – SmallChess Apr 10 '17 at 23:00