What do the terms "dense" and "sparse" mean in the context of neural networks (NNs)? What is the difference between them? Why are they so called?


In mathematics, "sparse" and "dense" often refer to the number of zero vs. non-zero elements in an array (e.g. vector or matrix). A sparse array is one that contains mostly zeros and few non-zero entries. A dense array contains mostly non-zeros.

There's no hard threshold for what counts as sparse; it's a loose term, but can be made more specific. For example, a vector is $k$-sparse if it contains at most $k$ non-zero entries. Another way of saying this is that the vector's $\ell_0$ norm is $k$.

The usage of these terms in the context of neural networks is similar to their usage in other fields. In the context of NNs, things that may be described as sparse or dense include the activations of units within a particular layer, the weights, and the data. One could also talk about "sparse connectivity", which refers to the situation where only a small subset of units are connected to each other. This is a similar concept to sparse weights, because a connection with zero weight is effectively unconnected.

"Sparse array" can also refer to a class of data types that are efficient for representing arrays that are sparse. This is a concept within the domain of programming languages. It's related to, but distinct from the mathematical concept.

  • 2
    $\begingroup$ I think it would be useful to give an example of the usage of "sparse" and "dense" in the context of NNs. You could e.g. mention some NN architecture which uses both dense and sparse layers to do some task. You could explain why both dense and sparse layers are used. I think this would definitively improve the clarity and completeness of your answer. $\endgroup$ – nbro Oct 2 '18 at 11:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.