5
$\begingroup$

I am currently try to use auto-encoders based deep learning for my classification problem. I have found some very nice examples from Matlab website auto-encoders example.

  1. In this example, they used 2 auto-encoders layers and a softmax layer (see the figure below). Will the deep learning community consider this structure as a 'deep' or still a 'shallow' network? I am asking because it seems to me that in recent literatures, there are many deep networks contains >10 layers.

AE structure

  1. May I ask if in general 'deeper' network will produce better classification? Or if there are any tricks to improve the classification performance for auto-encoders?
$\endgroup$
2
$\begingroup$

A two-layer neural network is often considered a "deep learning model," but it's not especially deep. The 19-layer VGG-19 model is definitely deep. It's a continuum, and how many layers your model should have is a complex function of how much training data you have, how able to optimize it you are, and how hard the problem is.

With deep networks, there are two distinct problems going on:

  1. Does there exist a network with the architecture I'm considering that can solve my problem?
  2. Will I be able to find that network with typical optimization techniques given the data I have?

Roughly speaking, making a network deeper makes problem 1 easier but problem 2 harder. Still speaking very broadly, a 20-layer deep net can do anything a 2-layer one can, and more, but given your limited training data and time to run your optimization procedure to train it, you may be able to find a good 2-layer network relatively quickly while the 20-layer network is just floundering around.

Or if there are any tricks to improve the classification performance for auto-encoders?

This is far too broad a question for this format.

$\endgroup$
1
$\begingroup$

"Deep" is a marketing term: you can therefore use it whenever you need to market your multi-layered neural network.

More: Minimum number of layers in a deep neural network

$\endgroup$
1
$\begingroup$

Agree with @Franck Dernoncourt, I think "deep" or "not deep" doesn't matter. What matters is that if this model works on your task. For example, in the natural language procesing, modelling one sentence usually uses only one convolutional layer while in image recognition, they use a thousand layers. However you can not just simply apply more layers on different tasks.

$\endgroup$
  • $\begingroup$ "while in image recognition, they use a thousand layers" – I definitely haven't seen any thousand-layer models. :) As far as I know (not exactly my area), they deepest common model in use in image recognition is ~30. $\endgroup$ – Dougal Sep 17 '16 at 3:55
  • $\begingroup$ arxiv.org/pdf/1512.03385v1.pdf Hi I saw the a thousand layer from this paper, although the a thousand layers is not better then merely a hundred layers from their experiments $\endgroup$ – hidemyname Sep 17 '16 at 8:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.