What do the terms "dense" and "sparse" mean in the context of neural networks (NNs)? What is the difference between them? Why are they so called?
In mathematics, "sparse" and "dense" often refer to the number of zero vs. non-zero elements in an array (e.g. vector or matrix). A sparse array is one that contains mostly zeros and few non-zero entries. A dense array contains mostly non-zeros.
There's no hard threshold for what counts as sparse; it's a loose term, but can be made more specific. For example, a vector is $k$-sparse if it contains at most $k$ non-zero entries. Another way of saying this is that the vector's $\ell_0$ norm is $k$.
The usage of these terms in the context of neural networks is similar to their usage in other fields. In the context of NNs, things that may be described as sparse or dense include the activations of units within a particular layer, the weights, and the data. One could also talk about "sparse connectivity", which refers to the situation where only a small subset of units are connected to each other. This is a similar concept to sparse weights, because a connection with zero weight is effectively unconnected.
"Sparse array" can also refer to a class of data types that are efficient for representing arrays that are sparse. This is a concept within the domain of programming languages. It's related to, but distinct from the mathematical concept.