I am at the moment trying to build an image classifier, capable of determining if an image contains an object X.

To do this I have been thinking of using deep learning, to make the system more autonomous, and make it less humanly preprocessed as possible..

I using the caffe library to create my classifier, and tried different training which all have ended up giving me a declining learning rate proportional to the amount of iteration i run.

I tried tweaking at the different parameter, but the only one i haven't tweaked at is the network layer structure.

I have been using one which has been used for the classification of MNIST dataset, solely because i haven't been sure how to tackle the issue, of defining it, and how I should create it for my purpose.. Which is why I am asking you, what kind of approaches do you use for defining the layers for deep learning project.. How many, what layer, what type? and so on...