I am studying the effect of micro vs macro averaging the results of a cross-validation run. Using true positive rate as a running example, given a cross-validation run of $K$ folds, the final $TPR$ can be computed by:

  • micro-averaging: $$TPR_\mu = \dfrac{\sum_{i=1}^K TP_i}{\sum_{i=1}^K N_i^+}=\dfrac{\sum_{i=1}^K TP_i}{N^+}$$
  • macro-averaging: $$TPR_M=\dfrac1K\sum_{i=1}^K \dfrac{TP_i}{N_i^+}$$ where $N_i^+$ is the number of positive examples in fold $i$ and $N^+$ is the total number of positive examples.

Other answers and literature [1,2] argue that $TPR_\mu$ is the most unbiased estimator for the $TPR$. Here, I am interested in quantifying the difference between $TPR_\mu$ and $TPR_M$, particularly with respect to the number of positive examples in each fold, to better understand the sources of a potential bias in $TPR_M$.

My attempt is the following:

Let $f_i=\dfrac{N_i^+}{N_i}$ be the fraction of positives in fold $i$, and $f = \dfrac{N^+}{N}$ be the fraction of positives in the entire dataset. We have that: $$ \begin{align} N_i^+ &= f_i \cdot N_i = f_i \cdot \dfrac{N}{K} \\ N_i^+ &\sim \mathsf{Binomial} \left( \frac{N}{K}, \ f \right) \\ f_i &\sim \mathcal{N} \left( f, \ \sqrt{\frac{K}{N} \cdot f \cdot (1-f)} \right) \end{align} $$

Finally, $$ \begin{align} TPR_\mu &= \dfrac{\sum_{i=1}^K TP_i}{\sum_{i=1}^K N_i^+} = \dfrac{1}{N}\sum_{i=1}^K \color{blue}{\dfrac{TP_i}{f}} \\ TPR_M &= \dfrac1K\sum_{i=1}^K \dfrac{TP_i}{N_i^+}=\dfrac1K\sum_{i=1}^K \dfrac{TP_i}{f_i\cdot\frac{N}{K}} = \dfrac{1}{N}\sum_{i=1}^K\color{magenta}{\dfrac{TP_i}{f_i}} \end{align} $$

So, essentially, the difference between micro and macro averaging is solely due to the fraction of positives in each split. If each split contains the same fraction of positives (i.e. $f_i = f, \ \forall i$), then $TPR_\mu = TPR_M$.

My questions:

  1. Can I quantify some sort of difference between $TPR_\mu$ and $TPR_M$ given the variance of $f_i$? For example, if the variance is high, I expect $TPR_M$ to greatly differ from $TPR_\mu$, whereas if all $f_i$ are close to the mean, there wouldn't be much difference.
  2. Are there any interesting insights I can extract from this calculations?

[1] https://www.kdd.org/exploration_files/v12-1-p49-forman-sigkdd.pdf

[2] Mean(scores) vs Score(concatenation) in cross validation



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.