AlexNet implementation in Tensorflow not converging, huge loss

Question

I implemented the AlexNet Oxford 17 Flowers example from the tensorflow API tflearn using the CIFAR10 source code from TensorFlow. Like described in the paper of Alex Krizhevsky ("ImageNet Classification with Deep Convolutional Neural Networks"), I am using five convolutional layers with max pooling followed by 3 fully connected layers.

Using tflearn, I got this example working. Unfortunately there is an issue with saving and loading the data so I implemented the AlexNet with TensorFlow leaving tflearn behind.

Now, when starting the learning process, I get a initial loss value of ~169. After about 10000 steps, each with a batch size of 32 and a learning rate of 0.1, the loss value drops to ~0.57 what seemed like its working. Unfortunately, consulting TensorBoard I get strange behavior of all values like loss:

I played a lot with all the values but it doesn't seem to get any better. Already the high initial loss value makes me wonder, I thought it should be the -ln() of the probability of my classes using cross entropy. I just want to do binary classification (like soccer ball in picture or no soccer ball in picture) what results for me in an initial loss of about 0.69 (that also works in the tflearn example for me).

Has someone of you an idea what is causing that way too high loss value and if this is the reason my network isn't converging (or the other way round)?

Answer 1

I think you may want to reduce your learning rate to around 0.001. High learning rates can take make you vary a lot or even diverge along the error surface.

I myself have been implementing tensor flow alexnet for the past week, so I did a quick test on my data

With learning rate the initial error rate without any training: around 2.0 for either learning rate

After first batch is trained: 62000 for learning rate 0.1, 1.8 for learning rate 0.001

After around 2000 batches are trained loss comes back to around 2.0 for learning rate 0.1