# Loss and dropout in deep learning [duplicate]

I have a CNN with 3 convolutional layers, 1 max-pooling layer and 2 fully-connected layers before applying softmax classification. The CNN is trained with Adagrad and I achieve a quite good performance. However, I'm curious as to why my loss is incredibly very stochastic (see below). Throughout the 30000 iterations it will actually jump above the initial loss. Despite that, the accuracy of the CNN is pretty consistent throughout training. Could this be due to the use of dropout on the convolutional layers and fully-connected layers? If so, where is this mentioned in any scientific articles, lecture notes or tutorials?

Edit: Added parameters below. The learning rate and weight decay is found using gridsearch and different values doesn't change the loss much. It might be that I haven't tuned them completely correct though. Actually, I'm surprised how sensitive Adagrad is to these variables (increasing LR/WD by a factor 10 from these values actually causes my learning to diverge). I'll look into adding a validation/test loss when I have time.

• Learning Rate: 0.003
• Weight Decay: 0.0005
• Dropout: 0.5
• Minibatch-size: 10

## marked as duplicate by Sycorax, Michael Chernick, kjetil b halvorsen, gui11aume, mdeweyJul 5 '18 at 17:23

• In order to check your model, I would suggest if you could you print the validation or test loss too. Make sure that when you evaluate both of them you turn off dropout. Dropout should only be used in training. If you are using Torch there is an option model:evaluate() and model:training() to switch between modes. What is your dropout ratio? Although it doesn't really affect adagrad, but have you tried changing the initial learning rate (e.g. 1e-1, 1e-2, 1e-3)? Finally, what is your mini-batch size? – Yannis Assael May 29 '15 at 15:03