I'm using Caffe to train a CNN on a fairly large image set at work. It's about 200K images consisting of profile photos people upload to our app. The labels are either "approve" or "reject" based on guidelines by our community (i.e. no sexual organs, lude finger gestures, etc.). My strategy has been to take a pre-trained model from the Caffe Model Zoo (bvlc_reference_caffenet) and start from there (similar to what is done in the flickr fine tuning example in the example director). No matter what I have tried (wide range of learning rates from 0.1 to 1e-9, freezing several convolution layers, training from scratch, etc), nothing seems to work. The model basically gets to 50% so it's just like flipping a coin (the two classes are balanced 50/50 in the training and test sets). So it's pretty clear the model never actually learns anything.
Furthermore, when I train from scratch, I'm not able to reproduce even the very low-level convolution layers where you normally get edge filters, etc. It's odd to me because my data set isn't too far off from the ImageNet data set which seems to be mostly natural things.
I'm really excited about deep learning, but it's definitely not as simple as I had estimated from the enthusiastic blog posts on it! Seems like more art than science to me right now.