I'm studying this Tutorial on Variational Autoencoders by Carl Doersch. In the second page it states:

One of the most popular such frameworks is the Variational Autoencoder [1, 3], the subject of this tutorial. The assumptions of this model are weak, and training is fast via backpropagation. VAEs do make an approximation, but the error introduced by this approximation is arguably small given high-capacity models. These characteristics have contributed to a quick rise in their popularity.

I've read in the past these sort of claims about high-capacity models, but I don't seem to find any clear definition for it. I also found this related stackoverflow question but to me the answer is very unsatisfying.

Is there a definition for the capacity of a model? Can you measure it?


Capacity is an informal term. It's very close (if not a synonym) for model complexity. It's a way to talk about how complicated a pattern or relationship a model can express. You could expect a model with higher capacity to be able to model more relationships between more variables than a model with a lower capacity.

Drawing an analogy from the colloquial definition of capacity, you can think of it as the ability of a model to learn from more and more data, until it's been completely "filled up" with information.

There are various ways to formalize capacity and compute a numerical value for it, but importantly these are just some possible "operationalizations" of capacity (in much the same way that, if someone came up with a formula to compute beauty, you would realize that the formula is just one fallible interpretation of beauty).

VC dimension is a mathematically rigorous formulation of capacity. However, there can be a large gap between the VC dimension of a model and the model's actual ability to fit the data. Even though knowing the VC dim gives a bound on the generalization error of the model, this is usually too loose to be useful with neural networks.

Another line of research see here is to use the spectral norm of the weight matrices in a neural network as a measure of capacity. One way to understand this is that the spectral norm bounds the Lipschitz constant of the network.

The most common way to estimate the capacity of a model is to count the number of parameters. The more parameters, the higher the capacity in general. Of course, often a smaller network learns to model more complex data better than a larger network, so this measure is also far from perfect.

Another way to measure capacity might be to train your model with random labels (Neyshabur et. al) -- if your network can correctly remember a bunch of inputs along with random labels, it essentially shows that the model has the ability to remember all those data points individually. The more input/output pairs which can be "learned", the higher the capacity.

Adapting this to an auto-encoder, you might generate random inputs, train the network to reconstruct them, and then count how many random inputs you can successfully reconstruct with less than $\epsilon$ error.

  • 1
    $\begingroup$ This is a better answer than the one from stackoverflow, thank you. I still have some trouble with seeing loosely defined terms used as a justification for something else, but I guess that's how the field is moving forward. $\endgroup$ Nov 9 '17 at 13:41
  • 2
    $\begingroup$ i think misleading is a bit of a strong word here. i didn't know until now that there are decent bounds on VC dim as a function of params, but afaict from skimming abstracts these are still fairly loose, and only apply under fairly restrictive conditions, so i think what i said is still mostly true. you can use #params to describe the capacity of a GAN, but afaik VC dimension can't be applied to generative models. anyway, i think we should agree to disagree, and if you want to write an answer to this question centered on vc dimension, i'd love to read and learn from it. $\endgroup$
    – shimao
    Nov 22 '20 at 4:07
  • $\begingroup$ This is a good answer. Capacity is not the same as parameters because then it would be the same as degrees of freedom. Something weird about how there's a limited number of parameter values and optimization paths in a neural network. Whereas a linear model will always find an optimal solution (for that structure) and the parameters can be anything. $\endgroup$
    – grofte
    Jan 25 at 14:36

In the fundamental challenge of Machine Learning: Does the model I built truly generalize? one way to approach it is by using model capacity.

Higher the model capacity, the more expressive the model (i.e., it can accommodate more variation).

Capacity needs to be tuned with respect to the amount of data at hand. If a dataset is small we are better off training models with lower capacity.

Yes you can measure the capacity of a model if you first agree what a good metric has to be used. Usually variation is used for that.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.