Measuring multi-classification error - Coverage

Question

As a part of going through the newly release TidyModels with R by Max Kuhn and Julia Silge I dug into some of the references on Chapter 9. More specifically A Unified View of Multi-Label Performance Measures and while making certain I understood the notation, I quickly hit a wall on several of the measurements.

To keep my question focused, I will only ask about one of the measurements mentioned in the article here, coverage.

The article defines the coverage measure of a multi-classification problem is defined as
$$ coverage(F) = \frac{1}{m}\sum_{i=1}^{m}[\![\underset{j\in Y_{i.}^{+}}{\mathrm{max}}\ \mathrm{rank}_F(x_i,j) - 1]\!] $$ where

• $x_i\in \mathbb{R}^{d\times 1}$ is a real valued vector
• $Y_{i.}^{+} = \{j|y_{ij} = 1\}$ denotes the index set of relevant observations
• $y_{ij}\in\{0, 1\}^{l\times 1}$ is a label vector
• $F\ :\ \mathbb{R}^{d}\rightarrow \mathbb{R}^{l} = \{f_1, \ldots , f_l\}$ is the multi-label predictor function mapping a probability value for $x_i$ for label $y_{ij}$
• $m$ is the number of observations
• $l$ is the number of unique labels in $Y$ where $Y_i = \{y_{i1},\ldots, y_{il}\}$.

and gives a confusing description of the measure as

The number of more labels on aver-age should include to cover all relevant labels

My question becomes how to interpret the above equation, and given a defunct example where $m=1$ how to implement the formula based on their definition. Hopefuly the latter is glaringly obvious given the first.

Defunc example: $m=1$

From the modeldata package I'll use the first wrongly specified label in the hpc_cv dataset

library(modeldata)
library(dplyr)
data('hpc_cv', package = 'modeldata')
filter(hpc_cv, obs != pred) %>% select(obs, pred, VF, F, M, L) %>% head(1)
  obs pred        VF         F          M           L
1  VF    F 0.3761772 0.5456339 0.07679576 0.001393113

Repeating the question: How should the above formulation of coverage be interpreted and given the example data above what would the result become.

Answer 1

After letting the question simmer a bit further (an embarrassingly long time lets be honest) I've come to the conclusion that the answer is simple.

coverage definition

The coverage definition is $$ coverage(F) = \frac{1}{m}\sum_{i=1}^{m}[\![\underset{j\in Y_{i.}^{+}}{\mathrm{max}}\ \mathrm{rank}_F(x_i,j) - 1]\!] $$ What confused me in this definition was the odd way the authors wrote the formula. It became more clear when I rethought the problem using indicator functions. First of all the entire thing comes down to rewriting the expression $\underset{j\in Y_{i.}^{+}}{\mathrm{max}}\ \mathrm{rank}_F(x_i,j) - 1$. What the function "tries" to state states is

Compute the rank of $F(x_i)$. Of the rank of $F(x_i)$ where $y_{ij} \in Y_{i.}^+$ compute the maximum of the argument and $l$.

While I got confused by the function body stating $j\in Y_{i.}^+$, the intuitive answer is that $Y_{i.}^+$ is the set of indices of the relevant class labels and the maximum index is $l$. It is however important to note that this is not exactly what the article states, as $max(j)|j\in Y_{i.}^{+}$ could be an empty value, as $Y_{i.}^+$ may be an empty set, so the definition of coverage only makes sense if it is defined $$ coverage(F) = \frac{1}{m}\sum_{i=1}^{m}[\![\underset{j\in Y_{i.}^{+}}{\mathrm{max}}\ \mathrm{rank}_F(x_i,l) - 1]\!] $$ although I could further see this as being confusing.

defunc example $m = 1$

Now with this cleared up manually calculating the example is simple. The problem only has a single correct label, so letting $F(x_i)$ be the probability function of our model. Ignoring knowledge of the specific dataset we now have 2 possibilities:

1. The outcome is labelled using a function $H(x_i) = \underset{j}{\mathrm{arg}\ \mathrm{max}} F(x_i)_j$
2. The outcome is labelled similarly using $H(x_i)$ where $h_j(x_i) = [\![f_j(x_i) > t(x_i)]\!]$ where $t(x_i)$ is some threshold function.

The first situation is not truly multi-labelled and we'll have a situation where $$ \underset{j\in Y_{i.}^{+}}{\mathrm{max}}\ \mathrm{rank}_F(x_i,l) - 1 \in \left\{\begin{matrix} l - 1 \\ 0 \end{matrix}\right. $$ as we can only ever have on categorized label, this is either fully correctly labelled, or we mislabelled the observation. In the example above $obs = VF$ but $y_i \in \{VF, F, M, L\}$. For a single labelling function we have that either $pred = VF$ (in which case the rank is 1) or $pred != VF$ (in which case the observation is not in $Y_{i.}^+$ so the set is empty and the result is $l-1$).
For the second scenario we may have set a threshold $h_j(x_i)$, lets say this is defined as $f_j(x_i) > 0.2$. In this case $Y_{i.}^+ = {VF}$ and $Y_{i.}^-= {F}$ and (loosely) $rank(F(x_i)) = \{ 1_F, 2_{VF} \}$ (using the subscript to illustrate the classes). Thus $$ \underset{j\in Y_{i.}^{+}}{\mathrm{max}}\ \mathrm{rank}_F(x_i,l) - 1 = 2_{VF} - 1 = 1 $$ and so is the coverage for this single observation example.