I have read and heard in several places that Deep Learning Networks take considerably longer to train than, say, support vector/kernel machines, random forests or boosting methods, but they can give better performance.

My question is, what is fundamentally different about deep learning networks in relation to other learning methods that explains this difference? What is it known about their training complexity?

For example, are DNN discriminative (in contrast to the other methods)? It looks like they could be considered generative since Wikipedia mentions:

Once sufficiently many layers have been learned the deep architecture may be used as a generative model by reproducing the data when sampling down the model (an "ancestral pass") from the top level feature activations.

I have also read on Wikipedia that DNNs can be pre-trained in an unsupervised manner. Is this a notable difference in relation to other methods. Is this pre-training step that common? But most importantly, why do DNNs take so long to train in comparison to other methods?

  • $\begingroup$ Could you please give a reference where you read that? I'd be interested on reading a study comparing computational complexity and performance of the different classifiers. $\endgroup$ – jpmuc Sep 26 '13 at 13:24
  • $\begingroup$ @juampa. All I have is informal references (like the ones I just added) and my own experience in training deep belief nets in relation to other methods such as SVMs random forests, etc. I would also be definitely interested in a study like the one you mentioned, and that's why I am asking this question. $\endgroup$ – Josh Sep 30 '13 at 14:53
  • 1
    $\begingroup$ A lot of the other methods have convex loss functions, NN's do not have convex loss functions. We can optimise convex functions "easily", non-convex is a different story. This is a factor. $\endgroup$ – power Mar 2 '15 at 13:28
  • $\begingroup$ @Josh, I am not sure about this, but maybe it is because DNNs are deep, i.e. They have a lot more parameters compared to the other methods you talk about. If you could bring more formal references, it would help. $\endgroup$ – Azrael Jun 28 '15 at 8:44

Two qualitative answers that seem reasonable are that:

  • the more layers you have, the more computation you have to perform during a training step. This computation (think for example of backpropagation as the simplest example) may be linear with the number of layers, and with deep networks you easily get to 20-30 layers. Tools such as TensorFlow require to build a data flow graph in order for the computation to be parallelized while respecting dependencies between the single weight updates.
  • there are several hyperparameters of the network that have to be trained from data too. For example, consider choosing which filters to use in the first layer of a convolutional neural network: each combination that you try in an grid search or a random search results in a new network to train from scratch, and that has to be evaluated with all the others to find the one with the lowest error on a validation set.
| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.