What is the difference between a neural network and a deep neural network, and why do the deep ones work better?

I haven't seen the question stated precisely in these terms, and this is why I make a new question.

What I am interested in knowing is not the definition of a neural network, but understanding the actual difference with a deep neural network.

For more context: I know what a neural network is and how backpropagation works. I know that a DNN must have multiple hidden layers. However, 10 years ago in class I learned that having several layers or one layer (not counting the input and output layers) was equivalent in terms of the functions a neural network is able to represent (see Cybenko's Universal approximation theorem), and that having more layers made it more complex to analyse without gain in performance. Obviously, that is not the case anymore.

I suppose, maybe wrongly, that the differences are in terms of training algorithm and properties rather than structure, and therefore I would really appreciate if the answer could underline the reasons that made the move to DNN possible (e.g. mathematical proof or randomly playing with networks?) and desirable (e.g. speed of convergence?)

• Dec 25, 2015 at 17:47
• If the function you want to approximate is a compositional function (common in image processing and many natural domains due to the law of physics). It can be proved that deep networks can exploit this compositionality and achieve the same level of approximation error with exponentially less number of neurons (compared to a single-hidden-layer network). Ref: Poggio, Tomaso, et al. "Why and when can deep-but not shallow-networks avoid the curse of dimensionality: a review." International Journal of Automation and Computing (2017) Oct 5, 2018 at 5:45
• You may want to take a look at this article Apr 14, 2019 at 11:41
• See 6.4 of arxiv.org/pdf/2004.06093.pdf Apr 22, 2020 at 21:27

Let's start with a triviliaty: Deep neural network is simply a feedforward network with many hidden layers.

This is more or less all there is to say about the definition. Neural networks can be recurrent or feedforward; feedforward ones do not have any loops in their graph and can be organized in layers. If there are "many" layers, then we say that the network is deep.

How many layers does a network have to have in order to qualify as deep? There is no definite answer to this (it's a bit like asking how many grains make a heap), but usually having two or more hidden layers counts as deep. In contrast, a network with only a single hidden layer is conventionally called "shallow". I suspect that there will be some inflation going on here, and in ten years people might think that anything with less than, say, ten layers is shallow and suitable only for kindergarten exercises. Informally, "deep" suggests that the network is tough to handle.

Here is an illustration, adapted from here:

But the real question you are asking is, of course, Why would having many layers be beneficial?

I think that the somewhat astonishing answer is that nobody really knows. There are some common explanations that I will briefly review below, but none of them has been convincingly demonstrated to be true, and one cannot even be sure that having many layers is really beneficial.

I say that this is astonishing, because deep learning is massively popular, is breaking all the records (from image recognition, to playing Go, to automatic translation, etc.) every year, is getting used by the industry, etc. etc. And we are still not quite sure why it works so well.

I base my discussion on the Deep Learning book by Goodfellow, Bengio, and Courville which went out in 2017 and is widely considered to be the book on deep learning. (It's freely available online.) The relevant section is 6.4.1 Universal Approximation Properties and Depth.

You wrote that

10 years ago in class I learned that having several layers or one layer (not counting the input and output layers) was equivalent in terms of the functions a neural network is able to represent [...]

You must be referring to the so called Universal approximation theorem, proved by Cybenko in 1989 and generalized by various people in the 1990s. It basically says that a shallow neural network (with 1 hidden layer) can approximate any function, i.e. can in principle learn anything. This is true for various nonlinear activation functions, including rectified linear units that most neural networks are using today (the textbook references Leshno et al. 1993 for this result).

If so, then why is everybody using deep nets?

Well, a naive answer is that because they work better. Here is a figure from the Deep Learning book showing that it helps to have more layers in one particular task, but the same phenomenon is often observed across various tasks and domains:

We know that a shallow network could perform as good as the deeper ones. But it does not; and they usually do not. The question is --- why? Possible answers:

1. Maybe a shallow network would need more neurons then the deep one?
2. Maybe a shallow network is more difficult to train with our current algorithms (e.g. it has more nasty local minima, or the convergence rate is slower, or whatever)?
3. Maybe a shallow architecture does not fit to the kind of problems we are usually trying to solve (e.g. object recognition is a quintessential "deep", hierarchical process)?
4. Something else?

The Deep Learning book argues for bullet points #1 and #3. First, it argues that the number of units in a shallow network grows exponentially with task complexity. So in order to be useful a shallow network might need to be very big; possibly much bigger than a deep network. This is based on a number of papers proving that shallow networks would in some cases need exponentially many neurons; but whether e.g. MNIST classification or Go playing are such cases is not really clear. Second, the book says this:

Choosing a deep model encodes a very general belief that the function we want to learn should involve composition of several simpler functions. This can be interpreted from a representation learning point of view as saying that we believe the learning problem consists of discovering a set of underlying factors of variation that can in turn be described in terms of other, simpler underlying factors of variation.

I think the current "consensus" is that it's a combination of bullet points #1 and #3: for real-world tasks deep architecture are often beneficial and shallow architecture would be inefficient and require a lot more neurons for the same performance.

But it's far from proven. Consider e.g. Zagoruyko and Komodakis, 2016, Wide Residual Networks. Residual networks with 150+ layers appeared in 2015 and won various image recognition contests. This was a big success and looked like a compelling argument in favour of deepness; here is one figure from a presentation by the first author on the residual network paper (note that the time confusingly goes to the left here):

But the paper linked above shows that a "wide" residual network with "only" 16 layers can outperform "deep" ones with 150+ layers. If this is true, then the whole point of the above figure breaks down.

In this paper we provide empirical evidence that shallow nets are capable of learning the same function as deep nets, and in some cases with the same number of parameters as the deep nets. We do this by first training a state-of-the-art deep model, and then training a shallow model to mimic the deep model. The mimic model is trained using the model compression scheme described in the next section. Remarkably, with model compression we are able to train shallow nets to be as accurate as some deep models, even though we are not able to train these shallow nets to be as accurate as the deep nets when the shallow nets are trained directly on the original labeled training data. If a shallow net with the same number of parameters as a deep net can learn to mimic a deep net with high fidelity, then it is clear that the function learned by that deep net does not really have to be deep.

If true, this would mean that the correct explanation is rather my bullet #2, and not #1 or #3.

As I said --- nobody really knows for sure yet.

Concluding remarks

The amount of progress achieved in the deep learning over the last ~10 years is truly amazing, but most of this progress was achieved by trial and error, and we still lack very basic understanding about what exactly makes deep nets to work so well. Even the list of things that people consider to be crucial for setting up an effective deep network seems to change every couple of years.

The deep learning renaissance started in 2006 when Geoffrey Hinton (who had been working on neural networks for 20+ years without much interest from anybody) published a couple of breakthrough papers offering an effective way to train deep networks (Science paper, Neural computation paper). The trick was to use unsupervised pre-training before starting the gradient descent. These papers revolutionized the field, and for a couple of years people thought that unsupervised pre-training was the key.

Then in 2010 Martens showed that deep neural networks can be trained with second-order methods (so called Hessian-free methods) and can outperform networks trained with pre-training: Deep learning via Hessian-free optimization. Then in 2013 Sutskever et al. showed that stochastic gradient descent with some very clever tricks can outperform Hessian-free methods: On the importance of initialization and momentum in deep learning. Also, around 2010 people realized that using rectified linear units instead of sigmoid units makes a huge difference for gradient descent. Dropout appeared in 2014. Residual networks appeared in 2015. People keep coming up with more and more effective ways to train deep networks and what seemed like a key insight 10 years ago is often considered a nuisance today. All of that is largely driven by trial and error and there is little understanding of what makes some things work so well and some other things not. Training deep networks is like a big bag of tricks. Successful tricks are usually rationalized post factum.

We don't even know why deep networks reach a performance plateau; just 10 years people used to blame local minima, but the current thinking is that this is not the point (when the perfomance plateaus, the gradients tend to stay large). This is such a basic question about deep networks, and we don't even know this.

Update: This is more or less the subject of Ali Rahimi's NIPS 2017 talk on machine learning as alchemy: https://www.youtube.com/watch?v=Qi1Yry33TQE.

[This answer was entirely re-written in April 2017, so some of the comments below do not apply anymore.]

• Nice response! 1) As also mentioned by @Nicolas, it turns out that there is a theorem (en.wikipedia.org/wiki/Universal_approximation_theorem) that claims that a feed-forward neural network with a single hidden layer and a finite number of neurons can approximate any continuous function (including a DNN) on the unit hypercube. This result is claimed to be independent of the choice of the activation function. 2) I am not sure if your last remark (i.e. that Hinton joined Google) is relevant to the recent success of DNNs; Hinton joined Google long after DNNs became epidemic!
– Sobi
Dec 3, 2015 at 23:07
• Maybe we're using the terminology differently. Perceptrons don't have a hidden layer--their input is data, their output is the classification. A multilayer network consists of a bunch of perceptrons wired together such that the output from layer $n-1$ forms the input to layer $n$. A network with a single hidden layer thus has three layers (input, hidden, output). This hidden layer is the key for universal approximation: perceptrons, which lack it, can't compute things like XOR, but the multilayer networks can. Dec 4, 2015 at 18:13
• The linearity thing seems like a combination of two issues. Perceptrons can only compute linear decision boundaries--it can only draw straight lines to divide two classes. Multilayer networks can "draw" more complicated boundaries. But neither perceptrons nor multilayer networks use linear activation functions, except in the output layer of a multilayer network. The perceptron activation function is a heaviside (1 if x>0, 0 otherwise); multilayer networks often use sigmoids, but the constraints for universal approx. are pretty mild: non-constant, bounded, and monotonically increasing. Dec 4, 2015 at 18:23
• @amoeba great answer, the picture at the start kind of put me off to read the rest, but it was for the best. The perceptron is the counterpart of the linear regression for classification, which is why when people use the closed form solution (pseudoinverse) to solve the problem instead of online (sgd) methods, it's called Logistic Regression, because of the logistic (sigmoid function) = perceptron. A perceptron / logistic regression can only 'draw' linear decision boundaries and that's why it's called linear. Dec 6, 2015 at 4:42
• However, the multi-layer perceptron (what you drew in your first picture on the left) can combine multiple such linear decision boundaries and can thus partition the space to solve the (non-linear) XOR problem like @Matt mentioned. So, many many combined linear decision boundaries can make a circle for example if you squint your eyes. It depends on how you think about it - the decision is still linear in a higher space, if you are familiar with kernels, it's kind of the same thing. Dec 6, 2015 at 4:43

Good answer so far, though there are a couple of things nobody around here mentioned, here's my 0.02I'll just answer in the form of a story, should make things more fun and clear. No tldr here. In the process you should be able to understand what the difference is. There are multiple reasons why DNNs sparked when they did (stars had to align, like all things similar, it's just the matter of right place, right time etc). One reason is the availability of data, lots of data (labeled data). If you want to be able to generalize and learn something like 'generic priors' or 'universal priors' (aka the basic building blocks that can be re-used between tasks / applications) then you need lots of data. And wild data, might I add, not sterile data-sets carefully recorded in the lab with controlled lighting and all. Mechanical Turk made that (labeling) possible. Second, the possibility to train larger networks faster using GPUs made experimentation faster. ReLU units made things computationally faster as well and provided their regularization since you needed to use more units in one layer to be able to compress the same information since layers now were more sparse, so it also went nice with dropout. Also, they helped with an important problem that happens when you stack multiple layers. More about that later. Various multiple tricks that improved performance. Like using mini-batches (which is in fact detrimental for final error) or convolutions (which actually don't capture as much variance as local receptive fields) but are computationally faster. In the meantime people were debating if they liked em more skinny or more chubby, smaller or taller, with or without freckles, etc. Optimization was like does it fizz or does it bang so research was moving towards more complex methods of training like conjugate gradient and newtons method, finally they all realized there's no free lunch. Networks were burping. What slowed things down was the vanishing gradient problem. People went like: whoa, that's far out, man! In a nutshell it means that it was hard to adjust the error on layers closer to the inputs. As you add more layers on the cake, gets too wobbly. You couldn't back-propagate meaningful error back to the first layers. The more layers, the worse it got. Bummer. Some people figured out that using the cross-entropy as a loss function (well, again, classification and image recognition) provides some sort of regularization and helps against the network getting saturated and in turn the gradient wasn't able to hide that well. What also made things possible was the per-layer pre-training using unsupervised methods. Basically, you take a bunch of auto-encoders and learn increasingly less abstract representations as you increase the compression ratio. The weights from these networks were used to initialize the supervised version. This solved the vanishing gradient problem in another way: you're already starting supervised training from a much better start position. So all the other networks got up and started to revolt. But the networks needed supervision anyway, otherwise it was impossible to keep the big data still. Now, for the last part that finally sort of leads to your answer which is too complex to put in a nutshell: why more layers and not just one. Because we can! and because context and invariant feature descriptors. and pools. Here's an example: you have a data set of images, how are you going to train a plan NN using that data? Well, naively, you take let's say each row and you concatenate it into one long vector and that's your input. What do you learn? Well, some fuzzy nonsense functions that might not look like anything, because of the many many types of variances that the objects in the image contain and you are not able to distinguish between relevant and irrelevant things. And at some point the network needs to forget to be able to re-learn new stuff. So there's the capacity issue. This is more non-linear dynamics, but the intuition is that you need to increase the number of neurons to be able to include more information in your network. So the point is that if you just input the image as one piece, adding extra layers does not do too much for you since you're not able to learn abstractions, which is very important. Doing things holistically thus does not work that well, unless you're doing simpler things with the network like focusing on a specific type of object, so you limit yourself to one class and you pick on some global properties as a classification goal. So what's there to do? Look at the edge of your screen and try to read this text. Problem? As stupid as it sounds, you need to look at what you're reading. Otherwise it's too fuzzy / there's not enough resolution / granularity. Let's call the focus area the receptive field. Networks need to be able to focus too. Basically instead of using the whole image as input, you move a sliding window along the image and then you use that as input to the network (a bit less stochastic than what humans do). Now you also have a chance to capture correlations between pixels and hence objects and you also can distinguish between sleepy cat sitting on a sofa and an upside-down cat bungee jumping. Neat, faith in humanity restored. The network can learn local abstractions in an image on multiple levels. The network learns filters, initially simple ones and then builds up on those to learn more complex filters. So, to sum things up: receptive fields / convolutions, unsupervised initialization, rectified linear units, dropout or other regularization methods. If you're very serious about this I recommend you take a look at Schmidhuber's Deep Learning in Neural Networks: An Overview here's the url for the preprint http://arxiv.org/abs/1404.7828 And remember: big learning, deep data. Word. • Hi Florin, thanks for the nice answer! I like the writting style. When you speak about sliding windows, are you refering to how convolutional layers of convolutional NN observe different parts of an image and project their activations on a space of lower dimension? Dec 4, 2015 at 11:20 • pretty much yes, convolutions are not necessary, but they're faster computationally, since the weights are constrained. check out this paper where they don't use convolutions and use local receptive fields. the important keywords are local / hierarchical: arxiv.org/pdf/1112.6209.pdf Dec 5, 2015 at 11:00 • i also think the closest systematic answer is sobi's. he's got my upvote. i just added a few more things here and there with a bit of salt and pepper. Dec 5, 2015 at 11:04 In layman terms, the main difference with the classic Neural Networks is that they have much more hidden layers. The idea is to add labels to the layers to make several layers of abstraction: For example, a deep neural network for object recognition: • Layer 1: Single pixels • Layer 2: Edges • Layer 3: Forms(circles, squares) • Layer n: Whole object You can find a good explanation at this question in Quora. And, if you are interested in this subject I would reccoment to take a look at this book. • Thanks David, but I don't really see how to just add labels make it different. I also remember it was a hard problem to understand and decompose how a function was encoded byt the neural network. There must be something else than just having more layers. As for the exemple you gave, I believe that you can train individually (old school) neural networks to do each of the tasks. Nov 21, 2015 at 9:54 • The power comes when you use it like a pipeline, so the inputs and outputs from the layer stacks at every layer. Nov 21, 2015 at 10:06 • I have been reading your first link, which is a good ressource, and other related questions on quora and se, however the example you gave does not seems accurate with what I read. I'll try to answer my own question, summarizing theses readings. Nov 22, 2015 at 10:21 • Despite the pointed resource being interesting, the reply currenty does not answer the question. Dec 2, 2015 at 13:46 • Can you point to an example where the layers are labelled (other than in a purely descriptive way)? It is certainly true that hidden layers appear to successively more complex features in each hidden layer, but "adding labels" seems to imply that they're being specifically trained to do that. Apr 19, 2017 at 13:46 To expand on David Gasquez's answer, one of the main differences between deep neural networks and traditional neural networks is that we don't just use backpropagation for deep neural nets. Why? Because backpropagation trains later layers more efficiently than it trains earlier layers--as you go earlier and earlier in the network, the errors get smaller and more diffuse. So a ten-layer network will basically be seven layers of random weights followed by three layers of fitted weights, and do just as well as a three layer network. See here for more. So the conceptual breakthrough is treating the separate problems (the labeled layers) as separate problems--if we first try to solve the problem of building a generically good first layer, and then try to solve the problem of building a generically good second layer, eventually we'll have a deep feature space that we can feed in to our actual problem. NN: • one hidden layer is enough but can have multiple layers nevertheless, left to right ordering (model: feed forward NN) • trained only in supervised way (backpropagation) • when multiple layers are used, train all the layers at the same time (same algorithm: backpropagation), more layers makes it difficult to use as errors become too small • hard to understand what is learned at each layer DNN: • multiple layers are required, undirected edges (model: restricted boltzman machine) • first trained in an unsupervised way, where the networks learns relevant features by learning to reproduce its input, then trained in a supervised way that fines tune the features in order to classify • train the layers one by one from input to output layer (algorithm: contrastive divergence) • each layer clearly contains features of increasing abstraction The move to DNN is due to three independant breakthroughs which happened in 2006. Regarding theorems on NN, the one the question alludes to is: • universal approximation theorem or Cybenko theorem: a feed-forward neural network with a single hidden layer can approximate any continuous function. However in practice it may require much more neurons if a single hidden layer is used. • -1? Really? I read all this in literature and made a point by point comparison of both approaches! Please at least state what is not correct... Nov 30, 2015 at 15:41 • I did not downvote (perhaps a downvoter did not like that you answer your own question? but that is of course perfectly fine here), but here is one thing that is not entirely correct. What you list as properties of DNN: that edges are undirected, that it's first trained in an unsupervised way, that the layers are trained one by one -- all of that only refers to deep belief networks suggested by Hinton in 2006. This is not necessarily true for deep neural networks in general and in fact there are now many ways to train a deep network without all of that. See my answer. Dec 3, 2015 at 23:22 I've also been confused a bit in the beginning by the difference between neural networks (NN) and deep neural networks (DNN), however the 'depth' refers only to the number of parameters & layers, unfortunately. You can take it as some sort of re-branding under the so-called 'Canadian Mafia'. Several years ago, I also had Neural Networks as a part of a class and we did digit recognition, wave approximation and similar applications by using NN, which had multiple hidden layers and outputs and all that jazz that DNN's have. However, what we didn't have then was computing power. The reason that made the move to DNN possible and desirable are the advances in hardware development. Simply put, now we can compute more, faster and more parallelized (DNN on GPU's), while before, time was the bottleneck for NN's. As referenced on the Wikipedia's page for Deep Learning, the 'deep' part refers mostly to having features interact in a non-linear fashion on multiple layers, therefore performing feature extraction and transformation. This was also done in standard NN's, however at a smaller scale. On the same page, here you have the definition 'A deep neural network (DNN) is an artificial neural network (ANN) with multiple hidden layers of units between the input and output layers.' • Hi Mttk, thanks for your answer, it cast more light on this question. Regarding your last point, yes structuraly is very easy to explain the difference (1 vs multiple layers), but the difference in how these multiple layers are used seems to be what matters and it much less clear. That is why I focused the question not on structure.. Dec 1, 2015 at 12:59 • Honestly, I don't agree with the last definition I referenced - since ANN/NN with one hidden layer are not really efficient, and you needed multiple for any sort of more complex prediction (I'm saying this because I think DNN was a useless buzzword added to an already good term, NN). I think you can use NN and DNN interchangeably (because nowadays nobody uses one-hidden-layer NN's), while the usage of the layers differs between types of DNN's (CNN, RBM, RNN, LSTM, CW-RNN, ...) and not the idea of DNN itself. – mttk Dec 1, 2015 at 14:10 As far as I know, what is called Deep Neural Network (DNN) today has nothing fundamentally or philosophically different from the old standard Neural Network (NN). Although, in theory, one can approximate an arbitrary NN using a shallow NN with only one hidden layer, however, this does not mean that the two networks will perform similarly when trained using the same algorithm and training data. In fact there is a growing interest in training shallow networks that perform similarly to deep networks. The way this is done, however, is by training a deep network first, and then training the shallow network to imitate the final output (i.e. the output of the penultimate layer) of the deep network. See, what makes deep architectures favorable is that today's training techniques (back propagation) happen to work better when the neurons are laid out in a hierarchical structure. Another question that may be asked is: why Neural Networks (DNNs in particular) became so popular suddenly. To my understanding, the magic ingredients that made DNNs so popular recently are the following: A. Improved datasets and data processing capabilities 1. Large scale datasets with millions of diverse images became available 2. Fast GPU implementation was made available to public B. Improved training algorithms and network architectures 1. Rectified Linear Units (ReLU) instead of sigmoid or tanh 2. Deep network architectures evolved over the years A-1) Until very recently, at least in Computer Vision, we couldn't train models on millions of labeled images; simply because labeled datasets of that size did not exist. It turns out that, beside the number of images, the granularity of the label set is also a very crucial factor in the success of DNNs (see Figure 8 in this paper, by Azizpour et al.). A-2) A lot of engineering effort has gone into making it possible to train DNNs that work well in practice, most notably, the advent of GPU implementations. One of the first successful GPU implementations of DNNs, runs on two parallel GPUs; yet, it takes about a week to train a DNN on 1.2 million images of 1000 categories using high-end GPUs (see this paper, by Krizhevsky et al.). B-1) The use of simple Rectified Linear Units (ReLU) instead of sigmoid and tanh functions is probably the biggest building block in making training of DNNs possible. Note that both sigmoid and tanh functions have almost zero gradient almost everywhere, depending on how fast they transit from the low activation level to high; in the extreme case, when the transition is sudden, we get a step function that has slope zero everywhere except at one point where the transition happens. B-2) The story of how neural network architectures developed over the years reminds me of how evolution changes an organism's structure in nature. Parameter sharing (e.g. in convolutional layers), dropout regularization, initialization, learning rate schedule, spatial pooling, sub-sampling in the deeper layers, and many other tricks that are now considered standard in training DNNs were developed, evolved, end tailored over the years to make the training of the deep networks possible the way it is today. • +1. The first answer in this thread that provides an adequate reply to the OP's questions. Many good points here. My only major comment would be that in addition to your A and B, there is also C: Massive increase in the size of available training datasets. This seems to be at least as important as A and B. Dec 3, 2015 at 23:11 • I don't think relu is so important: Alex krizhevsky paper claimed it made nn learning up to 6 times faster. The most of the other network structure changes you mention relate to convolutional nns, which just copy standard image processing pipelines (good thing, but no new insights) Dec 3, 2015 at 23:19 • @amoeba: size of the dataset is under A. I updated the text to highlight it. – Sobi Dec 4, 2015 at 0:16 • @seanv507: indeed, I had convolutional networks (ConvNets) in mind when writing the answer. If there are other important factors (unrelated to ConvNets) that I have missed I would appreciate it if you mention them. I would be glad to update my answer accordingly. Regarding ReLUs, training nns with tanh and sigmoid is considerably harder than with ReLUs due to the issue of vanishing gradients: units get saturated easily and, once that happens, it is takes them long to become unsaturated again (gradients are very small when the unit is saturated) – Sobi Dec 4, 2015 at 0:17 The difference between a "Deep" NN and a standard NN is purely qualitative: there is no definition of what that "Deep" means. "Deep" can mean anything from the extremely sophisticated architectures that are used by Google, Facebook and co which have 50-80 or even more layers, to 2 hidden layers (4 layers total) architectures. I wouldn't be surprised if you could even find articles claiming to do deep learning with a single hidden layer, because "deep" doesn't mean much. "Neural network" is also a word that doesn't have a very precise meaning. It covers an extremely large ensemble of models, from random boltzman machines (which are undirected graphs) to feedforward architectures with various activation functions. Most NNs will be trained using backprop, but it doesn't have to be the case so even the training algorithms aren't very homogenous. Overall, deep learning, deep NNs and NNs have all become catch-all words which capture a multitude of approaches. For good introductory references into "what changed": Deep Learning of Representations: Looking Forward, Bengio, 2013 is a good review + perspective for the future. Also see Do Deep Nets Really Need to be Deep? Ba & Caruana, 2013 which illustrate that being deep might not be useful for representation but for learning. • The references you give are very helpful, but the rest of the answer in the current form (which reads as "NN does not mean anything, DNN does not mean anyhting, NN and DNN can do a lot of things") not much, would you consider revising it? Dec 1, 2015 at 12:08 I wouldn't say there is any big philosophical difference between NN and DNN (in fact I would say DNN is just a marketing term to distinguish from 'failed' NN) . What has changed is the size of the data sets. Essentially neural networks are currently the bestO(n)\$ statistical estimators, working well for high dimensional large datasets (e.g. imagenet).

I think you should step back and see that this has created a resurgence in shallow AI -- e.g. bag of words for sentiment analysis and other language applications and visual bag of words was leading approach to image recognition before DNN. No one is saying bag of words is a true model of language, but it is an effective engineering solution. So I would say DNN are a better 'visual bag of words' -- see e.g. Szegedy et al. 2013 Intriguing properties of neural networks and Nguyen et al. Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images where it is clear that there is no higher order structures etc. being learned (or whatever is claimed for DNN).

• @amoeba this other paper is almost a companion paper to the first (again with lots of images!) Dec 4, 2015 at 11:28

To answer the latter question, look at this paper from Telgarsky which says that for a certain classification problem "all shallow networks with fewer than exponentially (in k) many nodes exhibit error at least 1/6, whereas a deep network with 2 nodes in each of 2k layers achieves zero error."

The classification problem in question is the n-alternating-point problem in which we consider the interval $$[0,1-2^{-k}]$$ so that the input $$x_i$$ are the $$2^k$$ uniformly distributed points in that interval, and the corresponding $$y_i$$ are given by $$y_i=1$$ if $$i$$ is odd, and $$y_i=0$$ if $$i$$ is even. We then ask, how well can shallow networks without exponential widths capture this relationship in comparison to deep networks with just two nodes in each layer? Essentially, we can approximate data better (exactly even) with a linear (in $$k$$) number of layers with just two nodes in each layer, whereas we would need exponentially many (in $$k$$) nodes to get the same result in a shallow network.

The proof of the quotation involves noticing that the composition of non-linear activations applied to affine transformations (i.e. with a greater number of layers) manages to capture more variability in the data than summing those same functions (as in when we add nodes to layers).

Deep Learning is a set of algorithms in machine learning that attempt to model high-level abstractions in data by using architectures composed of multiple non-linear transformations.

Source: Arno Candel