As a part of going through the newly release TidyModels with R by Max Kuhn and Julia Silge I dug into some of the references on Chapter 9. More specifically A Unified View of Multi-Label Performance Measures and while making certain I understood the notation, I quickly hit a wall on several of the measurements.

To keep my question focused, I will only ask about one of the measurements mentioned in the article here, coverage.

The article defines the coverage measure of a multi-classification problem is defined as
$$ coverage(F) = \frac{1}{m}\sum_{i=1}^{m}[\![\underset{j\in Y_{i.}^{+}}{\mathrm{max}}\ \mathrm{rank}_F(x_i,j) - 1]\!] $$ where

  • $x_i\in \mathbb{R}^{d\times 1}$ is a real valued vector
  • $Y_{i.}^{+} = \{j|y_{ij} = 1\}$ denotes the index set of relevant observations
  • $y_{ij}\in\{0, 1\}^{l\times 1}$ is a label vector
  • $F\ :\ \mathbb{R}^{d}\rightarrow \mathbb{R}^{l} = \{f_1, \ldots , f_l\}$ is the multi-label predictor function mapping a probability value for $x_i$ for label $y_{ij}$
  • $m$ is the number of observations
  • $l$ is the number of unique labels in $Y$ where $Y_i = \{y_{i1},\ldots, y_{il}\}$.

and gives a confusing description of the measure as

The number of more labels on aver-age should include to cover all relevant labels

My question becomes how to interpret the above equation, and given a defunct example where $m=1$ how to implement the formula based on their definition. Hopefuly the latter is glaringly obvious given the first.

Defunc example: $m=1$

From the modeldata package I'll use the first wrongly specified label in the hpc_cv dataset

data('hpc_cv', package = 'modeldata')
filter(hpc_cv, obs != pred) %>% select(obs, pred, VF, F, M, L) %>% head(1)
  obs pred        VF         F          M           L
1  VF    F 0.3761772 0.5456339 0.07679576 0.001393113

Repeating the question: How should the above formulation of coverage be interpreted and given the example data above what would the result become.


1 Answer 1


After letting the question simmer a bit further (an embarrassingly long time lets be honest) I've come to the conclusion that the answer is simple.

coverage definition

The coverage definition is $$ coverage(F) = \frac{1}{m}\sum_{i=1}^{m}[\![\underset{j\in Y_{i.}^{+}}{\mathrm{max}}\ \mathrm{rank}_F(x_i,j) - 1]\!] $$ What confused me in this definition was the odd way the authors wrote the formula. It became more clear when I rethought the problem using indicator functions. First of all the entire thing comes down to rewriting the expression $\underset{j\in Y_{i.}^{+}}{\mathrm{max}}\ \mathrm{rank}_F(x_i,j) - 1$. What the function "tries" to state states is

Compute the rank of $F(x_i)$. Of the rank of $F(x_i)$ where $y_{ij} \in Y_{i.}^+$ compute the maximum of the argument and $l$.

While I got confused by the function body stating $j\in Y_{i.}^+$, the intuitive answer is that $Y_{i.}^+$ is the set of indices of the relevant class labels and the maximum index is $l$. It is however important to note that this is not exactly what the article states, as $max(j)|j\in Y_{i.}^{+}$ could be an empty value, as $Y_{i.}^+$ may be an empty set, so the definition of coverage only makes sense if it is defined $$ coverage(F) = \frac{1}{m}\sum_{i=1}^{m}[\![\underset{j\in Y_{i.}^{+}}{\mathrm{max}}\ \mathrm{rank}_F(x_i,l) - 1]\!] $$ although I could further see this as being confusing.

defunc example $m = 1$

Now with this cleared up manually calculating the example is simple. The problem only has a single correct label, so letting $F(x_i)$ be the probability function of our model. Ignoring knowledge of the specific dataset we now have 2 possibilities:

  1. The outcome is labelled using a function $H(x_i) = \underset{j}{\mathrm{arg}\ \mathrm{max}} F(x_i)_j$
  2. The outcome is labelled similarly using $H(x_i)$ where $h_j(x_i) = [\![f_j(x_i) > t(x_i)]\!]$ where $t(x_i)$ is some threshold function.

The first situation is not truly multi-labelled and we'll have a situation where $$ \underset{j\in Y_{i.}^{+}}{\mathrm{max}}\ \mathrm{rank}_F(x_i,l) - 1 \in \left\{\begin{matrix} l - 1 \\ 0 \end{matrix}\right. $$ as we can only ever have on categorized label, this is either fully correctly labelled, or we mislabelled the observation. In the example above $obs = VF$ but $y_i \in \{VF, F, M, L\}$. For a single labelling function we have that either $pred = VF$ (in which case the rank is 1) or $pred != VF$ (in which case the observation is not in $Y_{i.}^+$ so the set is empty and the result is $l-1$).
For the second scenario we may have set a threshold $h_j(x_i)$, lets say this is defined as $f_j(x_i) > 0.2$. In this case $Y_{i.}^+ = {VF}$ and $Y_{i.}^-= {F}$ and (loosely) $rank(F(x_i)) = \{ 1_F, 2_{VF} \}$ (using the subscript to illustrate the classes). Thus $$ \underset{j\in Y_{i.}^{+}}{\mathrm{max}}\ \mathrm{rank}_F(x_i,l) - 1 = 2_{VF} - 1 = 1 $$ and so is the coverage for this single observation example.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.