4
$\begingroup$

I am training a 3 layer neural network using back propagation (in python3). For that, I randomly initialize the weights in the network using numpy.random.randn(). I then use gradient descent to minimise the loss function. When I plot my cost function v/s the number of iterations, for different initial values of the weight matrix, I get different plots, looking like these. What could be a possible reason ? Can it be the case that there are two local minima in the vicinity, and based on the initial value of the weights, the descent step goes in different directions. Some times it's pretty good. Sometimes it's pretty smooth But sometimes it's very nasty, something like this : Sometimes it's pretty nasty The two labels being 0 and 1. The loss function is cross-entropy loss. Activation function is sigmoid on all nodes in the network.

$\endgroup$
1
  • $\begingroup$ Did you write the network code yourself? $\endgroup$
    – ABCD
    Apr 10, 2017 at 23:00

1 Answer 1

4
$\begingroup$

Craziness like after iteration ~150 on your second chart tends to be due to bad metaparameters (specifically the learning rate, or momentum if you're using it). The descent is overshooting convergence because the learning (or momentum) is too high. Bad regularization can amplify this effect as well. Cross-validation is typically used to tune them.

I'm not sure what's going on before iteration ~150 there (referring specifically to the four curves). It looks like there are multiple loss values for single iterations. If that is the case, the only way that could get plotted is if multiple loss values are being calculated and stored for single iterations.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.