How 'deep' is deep for auto-encoders based deep learning?

Question

I am currently try to use auto-encoders based deep learning for my classification problem. I have found some very nice examples from Matlab website auto-encoders example.

1. In this example, they used 2 auto-encoders layers and a softmax layer (see the figure below). Will the deep learning community consider this structure as a 'deep' or still a 'shallow' network? I am asking because it seems to me that in recent literatures, there are many deep networks contains >10 layers.

2. May I ask if in general 'deeper' network will produce better classification? Or if there are any tricks to improve the classification performance for auto-encoders?

Answer 1

A two-layer neural network is often considered a "deep learning model," but it's not especially deep. The 19-layer VGG-19 model is definitely deep. It's a continuum, and how many layers your model should have is a complex function of how much training data you have, how able to optimize it you are, and how hard the problem is.

With deep networks, there are two distinct problems going on:

1. Does there exist a network with the architecture I'm considering that can solve my problem?
2. Will I be able to find that network with typical optimization techniques given the data I have?

Roughly speaking, making a network deeper makes problem 1 easier but problem 2 harder. Still speaking very broadly, a 20-layer deep net can do anything a 2-layer one can, and more, but given your limited training data and time to run your optimization procedure to train it, you may be able to find a good 2-layer network relatively quickly while the 20-layer network is just floundering around.

Or if there are any tricks to improve the classification performance for auto-encoders?

This is far too broad a question for this format.

Answer 2

"Deep" is a marketing term: you can therefore use it whenever you need to market your multi-layered neural network.

More: Minimum number of layers in a deep neural network

Answer 3

Agree with @Franck Dernoncourt, I think "deep" or "not deep" doesn't matter. What matters is that if this model works on your task. For example, in the natural language procesing, modelling one sentence usually uses only one convolutional layer while in image recognition, they use a thousand layers. However you can not just simply apply more layers on different tasks.