Why is sparse categorical cross-entropy "sparse"? This is just a conceptual question... It seems to me that what is called categorical cross-entropy should be called sparse because with the one hot encoding it creates a sparse matrix/tensor (whereas actual sparse categorical cross-entropy creates a dense array). However, that is not how it is defined can anyone provide a conceptual framework to help me remember this?
 A: A vector of label indices can be viewed a specific kind representation of a sparse matrix.
An ordinary matrix stores all of the elements in it, so a $n \times p$ matrix stores $np$ values. But a sparse matrix only stores some of the  values, so a $n \times p$ sparse matrix could stores some number of values $k:0 \le k \le np$.
One way to store a sparse matrix is to record the row index $i$, the column index $j$, and the value $v$ in that location, so each entry is a tuple of three values, $(i,j,v)$. (Or, if you like, a dict {(i,j):v,...}.)
But the special case of sparse labels can improve on this. A matrix of one-hot vectors can be stored with $n$ values, instead of $3n$ values:

*

*a one-hot vector has only 1 nonzero entry;

*the value of the nonzero entry is 1.

Therefore, for a one-hot vectors, we only need to store the location of the nonzero value (the column $j$). We know each row has a single nonzero entry, so the position in the list tells us the row $i$. We know that the value of the nonzero entry must be 1.
On the other hand, the conventional way to store a matrix is to store all $np$ of the values, even if most of those values are zeros. In other words, the conventional method doesn't care about the fact that the matrix of labels is sparse and will naïvely store all of them. The sparse matrix exploits the structure of the labels to store fewer values.
