I implemented the AlexNet Oxford 17 Flowers example from the tensorflow API tflearn using the CIFAR10 source code from TensorFlow. Like described in the paper of Alex Krizhevsky ("ImageNet Classification with Deep Convolutional Neural Networks"), I am using five convolutional layers with max pooling followed by 3 fully connected layers.
Using tflearn, I got this example working. Unfortunately there is an issue with saving and loading the data so I implemented the AlexNet with TensorFlow leaving tflearn behind.
Now, when starting the learning process, I get a initial loss value of ~169. After about 10000 steps, each with a batch size of 32 and a learning rate of 0.1, the loss value drops to ~0.57 what seemed like its working. Unfortunately, consulting TensorBoard I get strange behavior of all values like loss:
I played a lot with all the values but it doesn't seem to get any better. Already the high initial loss value makes me wonder, I thought it should be the -ln() of the probability of my classes using cross entropy. I just want to do binary classification (like soccer ball in picture or no soccer ball in picture) what results for me in an initial loss of about 0.69 (that also works in the tflearn example for me).
Has someone of you an idea what is causing that way too high loss value and if this is the reason my network isn't converging (or the other way round)?