I'm currently trying to get the basics of Pytorch, playing around with simple networks topologies for the fashion-MNIST dataset.
You can find my code in this repo : https://github.com/sebastienwood/deeplearning-poly/blob/master/tp2/main.py
Moreover, the fact that the more "advanced" the networks I try to implement, the lossier it gets over the 50 epochs (Net4 should've been the conclusion of the learnings in the previous ones).
Interestingly, the accuracy of those nets seems to not reflect this loss going up. The 3rd networks performed the best with a final accuracy of around 92%, and the 2 firsts were around 90%. Last one got 89%.
I thought at first that it may be a problem of vanishing gradient, but using leaky relu activation function I thought this problem wouldn't appear. This is strange because the network 1 uses the sigmoid activation function and it outperform the leaky relu of network 2.
If anyone knows what is happening it would be a great help ! :)