What do the terms "dense" and "sparse" mean in the context of neural networks (NNs)? What is the difference between them? Why are they so called?
$\begingroup$
$\endgroup$
2
asked Mar 12, 2017 at 14:29
-
$\begingroup$ See also What does “sparse” mean in the context of neural nets? $\endgroup$– Scortchi ♦Commented Oct 2, 2018 at 11:20
-
$\begingroup$ Dense is saturated with non-zeros, Sparse comprises mane zero weights in matrix = Hence, sparce is difficult/slow to estimate, having lack of points & their dynamics in matrix/batches => need something like Convolution to eliminate zero-weights first (~regularization) to speed-up training, I think. Sparcity is the case of Images, computer vision, NLP, as is learning language models, cinema & phone-commutation signals, can use sparse-dence embeddings for better search results $\endgroup$– JeeyCiCommented May 30, 2023 at 6:26
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
0
In mathematics, "sparse" and "dense" often refer to the number of zero vs. non-zero elements in an array (e.g. vector or matrix). A sparse array is one that contains mostly zeros and few non-zero entries. A dense array contains mostly non-zeros.
There's no hard threshold for what counts as sparse; it's a loose term, but can be made more specific. For example, a vector is $k$-sparse if it contains at most $k$ non-zero entries. Another way of saying this is that the vector's $\ell_0$ norm is $k$.
The usage of these terms in the context of neural networks is similar to their usage in other fields. In the context of NNs, things that may be described as sparse or dense include the activations of units within a particular layer, the weights, and the data. One could also talk about "sparse connectivity", which refers to the situation where only a small subset of units are connected to each other. This is a similar concept to sparse weights, because a connection with zero weight is effectively unconnected.
"Sparse array" can also refer to a class of data types that are efficient for representing arrays that are sparse. This is a concept within the domain of programming languages. It's related to, but distinct from the mathematical concept.
answered Mar 12, 2017 at 16:41
$\begingroup$
$\endgroup$
Think of the term matrix as an image of a person's face. Sparse would mean that there are a few very distinct pixels in the image that carry a lot of meaning regarding the identity of a person. This could be e.g. the tip of your nose, the center of your pupils or the corners of your mouth. The sparse model would only use those pixels for distinguishing person A from person B. In comparison the term dense would imply that all the image pixels are evaluated to identify a person - e.g. by comparing the gradient of all adjacent pixels.
While sparse is usually more efficient because less data needs to be evaluated the dense model is more effective, but often times is too computation heavy for the task at hand.
In @user20160's answer the unused pixels would be the zeroes of my 2D image matrix.
