# Is there an alternative to categorical cross-entropy with a notion of “class distance”?

I have a signal $x \in \mathbb{R}^{t \times l}$ which is discretized into $l = 32$ levels for $t = 100000$ time points. This enables me to turn a regression problem into a classification problem, which is more tractable mathematically for my application.

I understand that classification problem should be done with categorical cross-entropy, and I realise that at any time point $t$, there is one level which is 1 In $x$, so probably sparse categorical cross-entropy would improve it.

However, in this problem setting, making $l = 15$ to 1 instead of $l = 16$ is not as bad as setting $l = 1$ to 1, as these levels have a natural order.

Is there any way to incorporate this information into the loss function?

I looked at the Wasserstein-distance metric, but I'm not reasonably advanced in mathematics to know if it has any closed from loss functions for my classes, but as far as I understand that would do something similar.

• 1) Is "level" a 1d variable before discretization? 2) What do your target values look like? E.g. for each data point, is there a single, correct discretized level, or a probability distribution of some sort? – user20160 Sep 20 '18 at 14:41
• I should have been more clear, but this is a one-hot encoding effectively. This is a single channel audio signal discretised in this case, so there is one single correct audio signal level for each time point. However, a bit higher and lower level would not be as bad as a lot higher and lot lower level. (Like in regression) – boomkin Sep 20 '18 at 16:06

## 1 Answer

The earth mover's distance (EMD) provides a way to do this. When computed between probability distributions, the EMD is equivalent to the 1st Wasserstein distance. Intuitively, each distribution can be imagined as a pile of dirt, consisting of a certain amount of dirt at each location. A pile can be transformed by moving the dirt from location to location. Work is measured as the amount of dirt moved times the distance moved. The EMD is defined as the minimum amount of work needed to transform one pile to match the other.

In your problem, there are multiple classes, each corresponding to one of the 'discretized levels'. Distances between classes are the distances between the corresponding levels. The classifier outputs a predicted probability that the input is a member of each class. In the dirt pile analogy, each class corresponds to a location, and the predicted probability defines the amount of dirt. For each point in the training set, you have a target class. This corresponds to a probability distribution that takes the value one for the target class and zero for all others, i.e. all dirt is piled up at a single location.

In general, computing the EMD requires solving an optimization problem, where we search over possible ways of transforming the dirt piles. But, the EMD has a convenient, closed form expression in your case, because there's only one transformation that makes sense: directly move all the dirt from wherever it was originally to a single target location. Suppose there are $$l$$ classes (represented as integers from 1 to $$l$$), and let $$D_{ij}$$ denote the distance between classes $$i$$ and $$j$$. For a given data point with target class $$c$$, let $$p_i$$ denote the classifier's predicted probability that the class is $$i$$. The EMD is:

$$\text{EMD}(p, c) = \sum_{i \ne c} p_i D_{ic}$$

Related references:

• Levina and Bickel (2001). The Earth Mover's Distance is the Mallows Distance: Some Insights from Statistics.
• Frogner et al. (2015). Learning with a Wasserstein Loss.
• Hou et al. (2017). Squared earth movers distance loss for training deep neural networks on ordered classes.