0
$\begingroup$

I'm currently trying to get the basics of Pytorch, playing around with simple networks topologies for the fashion-MNIST dataset.

However, when I record the loss of those models after each epochs, it seems its going up rather than going down. Here is a graph of those series : 50 epochs loss graph

You can find my code in this repo : https://github.com/sebastienwood/deeplearning-poly/blob/master/tp2/main.py

Moreover, the fact that the more "advanced" the networks I try to implement, the lossier it gets over the 50 epochs (Net4 should've been the conclusion of the learnings in the previous ones).

Interestingly, the accuracy of those nets seems to not reflect this loss going up. The 3rd networks performed the best with a final accuracy of around 92%, and the 2 firsts were around 90%. Last one got 89%.

I thought at first that it may be a problem of vanishing gradient, but using leaky relu activation function I thought this problem wouldn't appear. This is strange because the network 1 uses the sigmoid activation function and it outperform the leaky relu of network 2.

If anyone knows what is happening it would be a great help ! :)

Thanks

Improve this question
$\endgroup$
2
  • $\begingroup$ Is the plot training or validation error? $\endgroup$
    – Jan Kukacka
    Commented Feb 18, 2018 at 18:20
  • $\begingroup$ 50 first are training and the last is validation :) $\endgroup$
    – Seb
    Commented Feb 18, 2018 at 18:29

1 Answer 1

Reset to default
1
$\begingroup$

There may be many reasons for this behavior. I recommend checking this article which summarizes the most common issues while training neural networks and includes many debugging tips and tricks.

My guess is you should try different optimizer and/or smaller learning rate, which sometimes cause divergence.

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.