# Measuring multi-classification error - Coverage

As a part of going through the newly release TidyModels with R by Max Kuhn and Julia Silge I dug into some of the references on Chapter 9. More specifically A Unified View of Multi-Label Performance Measures and while making certain I understood the notation, I quickly hit a wall on several of the measurements.

To keep my question focused, I will only ask about one of the measurements mentioned in the article here, coverage.

The article defines the coverage measure of a multi-classification problem is defined as
$$coverage(F) = \frac{1}{m}\sum_{i=1}^{m}[\![\underset{j\in Y_{i.}^{+}}{\mathrm{max}}\ \mathrm{rank}_F(x_i,j) - 1]\!]$$ where

• $$x_i\in \mathbb{R}^{d\times 1}$$ is a real valued vector
• $$Y_{i.}^{+} = \{j|y_{ij} = 1\}$$ denotes the index set of relevant observations
• $$y_{ij}\in\{0, 1\}^{l\times 1}$$ is a label vector
• $$F\ :\ \mathbb{R}^{d}\rightarrow \mathbb{R}^{l} = \{f_1, \ldots , f_l\}$$ is the multi-label predictor function mapping a probability value for $$x_i$$ for label $$y_{ij}$$
• $$m$$ is the number of observations
• $$l$$ is the number of unique labels in $$Y$$ where $$Y_i = \{y_{i1},\ldots, y_{il}\}$$.

and gives a confusing description of the measure as

The number of more labels on aver-age should include to cover all relevant labels

My question becomes how to interpret the above equation, and given a defunct example where $$m=1$$ how to implement the formula based on their definition. Hopefuly the latter is glaringly obvious given the first.

### Defunc example: $$m=1$$

From the modeldata package I'll use the first wrongly specified label in the hpc_cv dataset

library(modeldata)
library(dplyr)
data('hpc_cv', package = 'modeldata')
filter(hpc_cv, obs != pred) %>% select(obs, pred, VF, F, M, L) %>% head(1)
obs pred        VF         F          M           L
1  VF    F 0.3761772 0.5456339 0.07679576 0.001393113


Repeating the question: How should the above formulation of coverage be interpreted and given the example data above what would the result become.

After letting the question simmer a bit further (an embarrassingly long time lets be honest) I've come to the conclusion that the answer is simple.

# coverage definition

The coverage definition is $$coverage(F) = \frac{1}{m}\sum_{i=1}^{m}[\![\underset{j\in Y_{i.}^{+}}{\mathrm{max}}\ \mathrm{rank}_F(x_i,j) - 1]\!]$$ What confused me in this definition was the odd way the authors wrote the formula. It became more clear when I rethought the problem using indicator functions. First of all the entire thing comes down to rewriting the expression $$\underset{j\in Y_{i.}^{+}}{\mathrm{max}}\ \mathrm{rank}_F(x_i,j) - 1$$. What the function "tries" to state states is

Compute the rank of $$F(x_i)$$. Of the rank of $$F(x_i)$$ where $$y_{ij} \in Y_{i.}^+$$ compute the maximum of the argument and $$l$$.

While I got confused by the function body stating $$j\in Y_{i.}^+$$, the intuitive answer is that $$Y_{i.}^+$$ is the set of indices of the relevant class labels and the maximum index is $$l$$. It is however important to note that this is not exactly what the article states, as $$max(j)|j\in Y_{i.}^{+}$$ could be an empty value, as $$Y_{i.}^+$$ may be an empty set, so the definition of coverage only makes sense if it is defined $$coverage(F) = \frac{1}{m}\sum_{i=1}^{m}[\![\underset{j\in Y_{i.}^{+}}{\mathrm{max}}\ \mathrm{rank}_F(x_i,l) - 1]\!]$$ although I could further see this as being confusing.

# defunc example $$m = 1$$

Now with this cleared up manually calculating the example is simple. The problem only has a single correct label, so letting $$F(x_i)$$ be the probability function of our model. Ignoring knowledge of the specific dataset we now have 2 possibilities:

1. The outcome is labelled using a function $$H(x_i) = \underset{j}{\mathrm{arg}\ \mathrm{max}} F(x_i)_j$$
2. The outcome is labelled similarly using $$H(x_i)$$ where $$h_j(x_i) = [\![f_j(x_i) > t(x_i)]\!]$$ where $$t(x_i)$$ is some threshold function.

The first situation is not truly multi-labelled and we'll have a situation where $$\underset{j\in Y_{i.}^{+}}{\mathrm{max}}\ \mathrm{rank}_F(x_i,l) - 1 \in \left\{\begin{matrix} l - 1 \\ 0 \end{matrix}\right.$$ as we can only ever have on categorized label, this is either fully correctly labelled, or we mislabelled the observation. In the example above $$obs = VF$$ but $$y_i \in \{VF, F, M, L\}$$. For a single labelling function we have that either $$pred = VF$$ (in which case the rank is 1) or $$pred != VF$$ (in which case the observation is not in $$Y_{i.}^+$$ so the set is empty and the result is $$l-1$$).
For the second scenario we may have set a threshold $$h_j(x_i)$$, lets say this is defined as $$f_j(x_i) > 0.2$$. In this case $$Y_{i.}^+ = {VF}$$ and $$Y_{i.}^-= {F}$$ and (loosely) $$rank(F(x_i)) = \{ 1_F, 2_{VF} \}$$ (using the subscript to illustrate the classes). Thus $$\underset{j\in Y_{i.}^{+}}{\mathrm{max}}\ \mathrm{rank}_F(x_i,l) - 1 = 2_{VF} - 1 = 1$$ and so is the coverage for this single observation example.