Why are neural networks becoming deeper, but not wider?

Question

In recent years, convolutional neural networks (or perhaps deep neural networks in general) have become deeper and deeper, with state-of-the-art networks going from 7 layers (AlexNet) to 1000 layers (Residual Nets) in the space of 4 years. The reason behind the boost in performance from a deeper network, is that a more complex, non-linear function can be learned. Given sufficient training data, this enables the networks to more easily discriminate between different classes.

However, the trend seems to not have followed with the number of parameters in each layer. For example, the number of feature maps in the convolutional layers, or the number of nodes in the fully-connected layers, has remained roughly the same and is still relatively small in magnitude, despite the large increase in the number of layers. From my intuition though, it would seem that increasing the number of parameters per layer would give each layer a richer source of data from which to learn its non-linear function; but this idea seems to have been overlooked in favour of simply adding more layers, each with a small number of parameters.

So whilst networks have become "deeper", they have not become "wider". Why is this?

Answer 1

As a disclaimer, I work on neural nets in my research, but I generally use relatively small, shallow neural nets rather than the really deep networks at the cutting edge of research you cite in your question. I am not an expert on the quirks and peculiarities of very deep networks and I will defer to someone who is.

First, in principle, there is no reason you need deep neural nets at all. A sufficiently wide neural network with just a single hidden layer can approximate any (reasonable) function given enough training data. There are, however, a few difficulties with using an extremely wide, shallow network. The main issue is that these very wide, shallow networks are very good at memorization, but not so good at generalization. So, if you train the network with every possible input value, a super wide network could eventually memorize the corresponding output value that you want. But that's not useful because for any practical application you won't have every possible input value to train with.

The advantage of multiple layers is that they can learn features at various levels of abstraction. For example, if you train a deep convolutional neural network to classify images, you will find that the first layer will train itself to recognize very basic things like edges, the next layer will train itself to recognize collections of edges such as shapes, the next layer will train itself to recognize collections of shapes like eyes or noses, and the next layer will learn even higher-order features like faces. Multiple layers are much better at generalizing because they learn all the intermediate features between the raw data and the high-level classification.

So that explains why you might use a deep network rather than a very wide but shallow network. But why not a very deep, very wide network? I think the answer there is that you want your network to be as small as possible to produce good results. As you increase the size of the network, you're really just introducing more parameters that your network needs to learn, and hence increasing the chances of overfitting. If you build a very wide, very deep network, you run the chance of each layer just memorizing what you want the output to be, and you end up with a neural network that fails to generalize to new data.

Aside from the specter of overfitting, the wider your network, the longer it will take to train. Deep networks already can be very computationally expensive to train, so there's a strong incentive to make them wide enough that they work well, but no wider.

Answer 2

I don't think there is a definite answer to your questions. But I think the conventional wisdom goes as following:

Basically, as the hypothesis space of a learning algorithm grows, the algorithm can learn richer and richer structures. But at the same time, the algorithm becomes more prone to overfitting and its generalization error is likely to increase.

So ultimately, for any given dataset, it's advisable to work with the minimal model that has enough capacity to learn the real structure of the data. But this is a very hand-wavy advice, since usually the "real structure of the data" is unknown, and often even the capacities of the candidate models are only vaguely understood.

When it comes to neural networks, the size of the hypothesis space is controlled by the number of parameters. And it seems that for a fixed number of parameters (or a fixed order of magnitude), going deeper allows the models to capture richer structures (e.g. this paper).

This may partially explain the success of deeper models with fewer parameters: VGGNet (from 2014) has 16 layers with ~140M parameters, while ResNet (from 2015) beat it with 152 layers but only ~2M parameters

(as a side, smaller models may be computationally easier to train - but I don't think it's a major factor by itself - since depth actually complicates the training)

Note that this trend (more depth, less parameters) is mostly present in vision-related tasks and convolutional networks, and this calls for a domain-specific explanation. So here's another perspective:

Each "neuron" in a convolutional layer has a "receptive field", which is the size and shape of the inputs that effects each output. Intuitively, each kernel captures some kind of a relation between nearby inputs. And small kernels (which are common and preferable) have a small receptive field, so they can provide information only regarding local relations.

But as you go deeper, the receptive field of each neuron with respect to a some earlier layer becomes larger. So deep layers can provide features with global semantic meaning and abstract details (relations of relations ... of relations of objects), while using only small kernels (which regularize the relations the network learns, and helps it converge and generalize).

So the usefulness of deep convolutional networks in computer vision may be partially explained by the spatial structure of images and videos. It's possible that time will tell that for different types of problems, or for non-convolutional architectures, depth actually doesn't work well.

Answer 3

Adding more features helps but the benefit quickly become marginal after a lot of features were added. That's one reason why tools like PCA work: a few components capture most variance in the features. Hence, adding more features after some point is almost useless.

On the other hand finding the right functional for ma of the feature is always a good idea. However, if you don't have a good theory it's hard to come up with a correct function, of course. So, adding layers is helpful as form of a brute force approach.

Consider a simple case: air drag of a car. Say, we didn't know the equation: $$f\sim C\rho A v^2/2$$ where $A$ - a crossectional area of a car, $\rho$ - air density, and $v$ - velocity of a car. We could figure that car measurements are important and add them as features, velocity of a car will go in too. So we keep adding features, and maybe add air pressure, temperature, length, width of a car, number of seats, etc.

We'll end up with a model like $$f\sim \sum_i\beta_i x_i$$ You see how these features are not going to assemble themselves into the "true" equation unless we add all interactions and polynomials. However, if the true equation wasn't conveniently polynomial, say it had exponents or other weird transcendental functions, then we'd have no chance to emulate it with expanding feature set or widening the network.

However, making the network deeper would easily get you to the equation above with just two layers. More complicated functions would need more layer, that's why deepening the number of layers could be a way to go in many problems.

Answer 4

For a densely connected neural net of depth $d$ and width $w$, the number of parameters (hence, RAM required to run or train the network) is $O(dw^2)$. Thus, if you only have a limited number of parameters, it often makes sense to prefer a large increase in depth over a small increase in width.

Why might you be trying to limit the number of parameters? A number of reasons:

• You are trying to avoid overfitting. (Although limiting the number of parameters is a very blunt instrument for achieving this.)
• Your research is more impressive if you can outperform someone else's model using the same number of parameters.
• Training your model is much easier if the model (plus moment params if you're using Adam) can fit inside the memory of a single GPU.
• In real life applications, RAM is often expensive when serving models. This is especially true for running models on e.g. a cell phone, but can sometimes apply even for serving models from the cloud.

Where does the $O(dw^2)$ come from? For two neighboring layers of width $w_1, w_2$, the connections between them are described by a $w_1 \times w_2$. So if you have $(d-2)$ layers of width $w$ (plus an input and an output layer), the number of parameters is $$(d-2) w^2 + w \cdot (\text{input layer width}) + w \cdot (\text{output layer width}) = O(dw^2)\text{.}$$ Instead of restricting the width, an alternate strategy sometimes used is to use sparse connections. For instance, when initializing the network topology, you can admit each connection with probability $1/\sqrt{w}$ so the total number of parameters is $O(dw)$. But if you do this, it's not clear that increasing the width will necessarily increase the model's capacity to learn.

Answer 5

I think you are get the in detail answer of the question through this paper name Impact of fully connected layers on performance of convolutional neural networks for image classification link - https://www.sciencedirect.com/science/article/pii/S0925231219313803.

It comes to the following conclusion -

In order to obtain better performance, the shallow CNNs require more nodes in FC layers. On the other hand, deeper CNNs need less number of neurons in FC layers irrespective of type of the dataset.

The shallow CNNs require a large number of neurons in FC layers as well as more number of FC layers for wider datasets compared to deeper datasets and vice-versa.

Deeper CNNs perform better than shallow models over deeper datasets. In contrast, shallow architectures perform better than deeper architectures for wider datasets. These observations can help the deep learning community while making a decision about the choice of deep/shallow CNN architectures.

Answer 6

Currently, on GPUs - we use 32-bit float and with 512 features - combining them we already get quite imprecise. Going even wider is hence limited by numerics and precision of 32-bit float. Another thing is that we actually should probably go wider if we care about accuracy: https://www.sciencedirect.com/science/article/pii/S2666827021000633