In recent years, convolutional neural networks (or perhaps deep neural networks in general) have become deeper and deeper, with state-of-the-art networks going from 7 layers (AlexNet) to 1000 layers (Residual Nets) in the space of 4 years. The reason behind the boost in performance from a deeper network, is that a more complex, non-linear function can be learned. Given sufficient training data, this enables the networks to more easily discriminate between different classes.

However, the trend seems to not have followed with the number of parameters in each layer. For example, the number of feature maps in the convolutional layers, or the number of nodes in the fully-connected layers, has remained roughly the same and is still relatively small in magnitude, despite the large increase in the number of layers. From my intuition though, it would seem that increasing the number of parameters per layer would give each layer a richer source of data from which to learn its non-linear function; but this idea seems to have been overlooked in favour of simply adding more layers, each with a small number of parameters.

So whilst networks have become "deeper", they have not become "wider". Why is this?


As a disclaimer, I work on neural nets in my research, but I generally use relatively small, shallow neural nets rather than the really deep networks at the cutting edge of research you cite in your question. I am not an expert on the quirks and peculiarities of very deep networks and I will defer to someone who is.

First, in principle, there is no reason you need deep neural nets at all. A sufficiently wide neural network with just a single hidden layer can approximate any (reasonable) function given enough training data. There are, however, a few difficulties with using an extremely wide, shallow network. The main issue is that these very wide, shallow networks are very good at memorization, but not so good at generalization. So, if you train the network with every possible input value, a super wide network could eventually memorize the corresponding output value that you want. But that's not useful because for any practical application you won't have every possible input value to train with.

The advantage of multiple layers is that they can learn features at various levels of abstraction. For example, if you train a deep convolutional neural network to classify images, you will find that the first layer will train itself to recognize very basic things like edges, the next layer will train itself to recognize collections of edges such as shapes, the next layer will train itself to recognize collections of shapes like eyes or noses, and the next layer will learn even higher-order features like faces. Multiple layers are much better at generalizing because they learn all the intermediate features between the raw data and the high-level classification.

So that explains why you might use a deep network rather than a very wide but shallow network. But why not a very deep, very wide network? I think the answer there is that you want your network to be as small as possible to produce good results. As you increase the size of the network, you're really just introducing more parameters that your network needs to learn, and hence increasing the chances of overfitting. If you build a very wide, very deep network, you run the chance of each layer just memorizing what you want the output to be, and you end up with a neural network that fails to generalize to new data.

Aside from the specter of overfitting, the wider your network, the longer it will take to train. Deep networks already can be very computationally expensive to train, so there's a strong incentive to make them wide enough that they work well, but no wider.

  • $\begingroup$ +1. Any comment on arxiv.org/abs/1605.07146? $\endgroup$ – amoeba Jul 13 '16 at 22:42
  • 2
    $\begingroup$ I only skimmed it, so I can't say anything authoritative, but it looks like the authors found that at least in the case of residual networks a wide (but still 16 layers deep!) net outperforms a narrow, extremely deep (1000 layers) net. I don't know much about residual networks, but according to the introduction it seems that a difficulty in training them is that there can be a tendency for layers to not learn anything at all and thereby not contribute much to the result. It seems that having fewer, but more powerful, layers avoids this. Whether this applies to other kinds of NNs I don't know. $\endgroup$ – J. O'Brien Antognini Jul 13 '16 at 23:12

I don't think there is a definite answer to your questions. But I think the conventional wisdom goes as following:

Basically, as the hypothesis space of a learning algorithm grows, the algorithm can learn richer and richer structures. But at the same time, the algorithm becomes more prone to overfitting and its generalization error is likely to increase.

So ultimately, for any given dataset, it's advisable to work with the minimal model that has enough capacity to learn the real structure of the data. But this is a very hand-wavy advice, since usually the "real structure of the data" is unknown, and often even the capacities of the candidate models are only vaguely understood.

When it comes to neural networks, the size of the hypothesis space is controlled by the number of parameters. And it seems that for a fixed number of parameters (or a fixed order of magnitude), going deeper allows the models to capture richer structures (e.g. this paper).

This may partially explain the success of deeper models with fewer parameters: VGGNet (from 2014) has 16 layers with ~140M parameters, while ResNet (from 2015) beat it with 152 layers but only ~2M parameters

(as a side, smaller models may be computationally easier to train - but I don't think it's a major factor by itself - since depth actually complicates the training)

Note that this trend (more depth, less parameters) is mostly present in vision-related tasks and convolutional networks, and this calls for a domain-specific explanation. So here's another perspective:

Each "neuron" in a convolutional layer has a "receptive field", which is the size and shape of the inputs that effects each output. Intuitively, each kernel captures some kind of a relation between nearby inputs. And small kernels (which are common and preferable) have a small receptive field, so they can provide information only regarding local relations.

But as you go deeper, the receptive field of each neuron with respect to a some earlier layer becomes larger. So deep layers can provide features with global semantic meaning and abstract details (relations of relations ... of relations of objects), while using only small kernels (which regularize the relations the network learns, and helps it converge and generalize).

So the usefulness of deep convolutional networks in computer vision may be partially explained by the spatial structure of images and videos. It's possible that time will tell that for different types of problems, or for non-convolutional architectures, depth actually doesn't work well.

  • $\begingroup$ The Restricted Boltzmann Machine part of deep networks is a large multiplier on convergence time. (afaict) $\endgroup$ – EngrStudent Jul 13 '16 at 21:10
  • 4
    $\begingroup$ RBMs are not inherent to deep learning. Many (nowadays, most?) successful deep networks don't use RBMs. $\endgroup$ – Borbei Jul 13 '16 at 21:18
  • 3
    $\begingroup$ Thanks for linking to that paper, I hadn't seen it before and it looks very relevant. $\endgroup$ – J. O'Brien Antognini Jul 13 '16 at 21:31
  • $\begingroup$ @Borbei - How do they assure separation of features without RBM? $\endgroup$ – EngrStudent Jul 13 '16 at 22:33
  • 1
    $\begingroup$ +1. Any comment on arxiv.org/abs/1605.07146? $\endgroup$ – amoeba Jul 13 '16 at 22:42

Adding more features helps but the benefit quickly become marginal after a lot of features were added. That's one reason why tools like PCA work: a few components capture most variance in the features. Hence, adding more features after some point is almost useless.

On the other hand finding the right functional for ma of the feature is always a good idea. However, if you don't have a good theory it's hard to come up with a correct function, of course. So, adding layers is helpful as form of a brute force approach.

Consider a simple case: air drag of a car. Say, we didn't know the equation: $$f\sim C\rho A v^2/2$$ where $A$ - a crossectional area of a car, $\rho$ - air density, and $v$ - velocity of a car. We could figure that car measurements are important and add them as features, velocity of a car will go in too. So we keep adding features, and maybe add air pressure, temperature, length, width of a car, number of seats, etc.

We'll end up with a model like $$f\sim \sum_i\beta_i x_i$$ You see how these features are not going to assemble themselves into the "true" equation unless we add all interactions and polynomials. However, if the true equation wasn't conveniently polynomial, say it had exponents or other weird transcendental functions, then we'd have no chance to emulate it with expanding feature set or widening the network.

However, making the network deeper would easily get you to the equation above with just two layers. More complicated functions would need more layer, that's why deepening the number of layers could be a way to go in many problems.

  • 5
    $\begingroup$ You assume linear transfer functions. But there are many other (common) choices, and according to the universal approximation theorem of ANNs, even a single hidden non-linear layer (if it's wide enough) can approximate any nice function. So representability can't really explain the success of deep networks. $\endgroup$ – Borbei Jul 13 '16 at 21:52
  • 2
    $\begingroup$ My example was linear but it applies to a wider set of cases. You assumed "nice" function but many are not so nice. For instance when i select a car to buy why would my decision algorithm to be a nice function? $\endgroup$ – Aksakal Jul 13 '16 at 22:08

For a densely connected neural net of depth $d$ and width $w$, the number of parameters (hence, RAM required to run or train the network) is $O(dw^2)$. Thus, if you only have a limited number of parameters, it often makes sense to prefer a large increase in depth over a small increase in width.

Why might you be trying to limit the number of parameters? A number of reasons:

  • You are trying to avoid overfitting. (Although limiting the number of parameters is a very blunt instrument for achieving this.)
  • Your research is more impressive if you can outperform someone else's model using the same number of parameters.
  • Training your model is much easier if the model (plus moment params if you're using Adam) can fit inside the memory of a single GPU.
  • In real life applications, RAM is often expensive when serving models. This is especially true for running models on e.g. a cell phone, but can sometimes apply even for serving models from the cloud.

Where does the $O(dw^2)$ come from? For two neighboring layers of width $w_1, w_2$, the connections between them are described by a $w_1 \times w_2$. So if you have $(d-2)$ layers of width $w$ (plus an input and an output layer), the number of parameters is $$(d-2) w^2 + w \cdot (\text{input layer width}) + w \cdot (\text{output layer width}) = O(dw^2)\text{.}$$ Instead of restricting the width, an alternate strategy sometimes used is to use sparse connections. For instance, when initializing the network topology, you can admit each connection with probability $1/\sqrt{w}$ so the total number of parameters is $O(dw)$. But if you do this, it's not clear that increasing the width will necessarily increase the model's capacity to learn.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.