What is the "capacity" of a machine learning model?

Question

I'm studying this Tutorial on Variational Autoencoders by Carl Doersch. In the second page it states:

One of the most popular such frameworks is the Variational Autoencoder [1, 3], the subject of this tutorial. The assumptions of this model are weak, and training is fast via backpropagation. VAEs do make an approximation, but the error introduced by this approximation is arguably small given high-capacity models. These characteristics have contributed to a quick rise in their popularity.

I've read in the past these sort of claims about high-capacity models, but I don't seem to find any clear definition for it. I also found this related stackoverflow question but to me the answer is very unsatisfying.

Is there a definition for the capacity of a model? Can you measure it?

Answer 1

Capacity is an informal term. It's very close (if not a synonym) for model complexity. It's a way to talk about how complicated a pattern or relationship a model can express. You could expect a model with higher capacity to be able to model more relationships between more variables than a model with a lower capacity.

Drawing an analogy from the colloquial definition of capacity, you can think of it as the ability of a model to learn from more and more data, until it's been completely "filled up" with information.

There are various ways to formalize capacity and compute a numerical value for it, but importantly these are just some possible "operationalizations" of capacity (in much the same way that, if someone came up with a formula to compute beauty, you would realize that the formula is just one fallible interpretation of beauty).

---

VC dimension is a mathematically rigorous formulation of capacity. However, there can be a large gap between the VC dimension of a model and the model's actual ability to fit the data. Even though knowing the VC dim gives a bound on the generalization error of the model, this is usually too loose to be useful with neural networks.

Another line of research see here is to use the spectral norm of the weight matrices in a neural network as a measure of capacity. One way to understand this is that the spectral norm bounds the Lipschitz constant of the network.

The most common way to estimate the capacity of a model is to count the number of parameters. The more parameters, the higher the capacity in general. Of course, often a smaller network learns to model more complex data better than a larger network, so this measure is also far from perfect.

Another way to measure capacity might be to train your model with random labels (Neyshabur et. al) -- if your network can correctly remember a bunch of inputs along with random labels, it essentially shows that the model has the ability to remember all those data points individually. The more input/output pairs which can be "learned", the higher the capacity.

Adapting this to an auto-encoder, you might generate random inputs, train the network to reconstruct them, and then count how many random inputs you can successfully reconstruct with less than $\epsilon$ error.

Answer 2

In the fundamental challenge of Machine Learning: Does the model I built truly generalize? one way to approach it is by using model capacity.

Higher the model capacity, the more expressive the model (i.e., it can accommodate more variation).

Capacity needs to be tuned with respect to the amount of data at hand. If a dataset is small we are better off training models with lower capacity.

Yes you can measure the capacity of a model if you first agree what a good metric has to be used. Usually variation is used for that.

Answer 3

This article from the blog Machine Learning Master is an interesting read to understand neural network's capacity: "How to Control Neural Network Model Capacity With Nodes and Layers".

A model with too little capacity cannot learn the training dataset meaning it will underfit, whereas a model with too much capacity may memorize the training dataset, meaning it will overfit or may get stuck or lost during the optimization process.

The capacity of a neural network model is defined by configuring the number of nodes and the number of layers.