Approaches used for defining deep layers?

Question

I am at the moment trying to build an image classifier, capable of determining if an image contains an object X.

To do this I have been thinking of using deep learning, to make the system more autonomous, and make it less humanly preprocessed as possible..

I using the caffe library to create my classifier, and tried different training which all have ended up giving me a declining learning rate proportional to the amount of iteration i run.

I tried tweaking at the different parameter, but the only one i haven't tweaked at is the network layer structure.

I have been using one which has been used for the classification of MNIST dataset, solely because i haven't been sure how to tackle the issue, of defining it, and how I should create it for my purpose.. Which is why I am asking you, what kind of approaches do you use for defining the layers for deep learning project.. How many, what layer, what type? and so on...

Answer 1

Architecture

Building deep neural net is a complex issue. There are no rules, which allow you to predict how many layers you should use, of what type and so on.

For image analysis, the rule of thumb is to use several convolutional (conv) / pooling (pool) layers at the bottom of the network (they work as feature extractors) and a few fully-connected (inner product, ip) layers at the top (they make classification based on features). The MNIST network is also organized in this way (conv1, pool1, conv2, pool2, ip1, relu, ip2).

To have idea about how networks like MNIST are constructed, you should read the paper:

Ciresan et al., "Flexible, High Performance Convolutional Neural Networks for Image Classification", IJCAI 2011

It describes one of the leading architectures for handwritten digit classification.

Domain adaptation

To train deep model you need a large dataset. Its size depends on variability of the object you need to detect. For real-world photography, state of the art is ImageNet with over 14 million images. Instead of building and training a model from scratch, you should consider domain adaptation (https://en.wikipedia.org/wiki/Domain_adaptation). The main idea is to use a trained model (not just architecture) and adapt it in supervised manner to detect object you need.

Learning rate

Learning rate decreases with number of iterations - its a parameter of stochastic gradient descent. Read more at Caffe webpage (http://caffe.berkeleyvision.org/tutorial/solver.html).