I have $N$ data points, each of which has an associated label that is either $0$ or $1$. For which data point I know the true label. Say I put all true labels in a vector $t$.

Next I ask Alice to label each data point (without access to the true labels) and collect her labels in $v^{(a)}$. I do the same with Bob to produce $v^{(b)}$.

By comparing $v^{(a)}$ with $t$, I can compute the f-score $f_a$ that shows how good Alice was at recognising the true labels. I can do the same for Bob to get $f_b$.

Question: assume $f_a$ and $f_b$ are different. How do I know if the difference is significant, or just occurred by chance?

What I've tried:

  • do an independent samples t-test between $v^{(a)}$ and $v^{(b)}$. The issue is that the $p$ values were over $0.05$ in most experiments, indicating insignificance, even when the values of $f_a$ and $f_b$ were massively different. I have the feeling that I should not be comparing these two vectors, but somehow compare their success in capturing the true labels.

  • so I tried an independent samples t-test $v^{(a)}\times t$ and $v^{(b)}\times t$ where "$\times$" denotes element-wise multiplication. The results looked better but not much so. I have the feeling that by comparing the sample "means", I do capture really what the f-score says and cannot therefore lead to pertinent conclusions about the difference in the two f-scores $f_a$ and $f_b$.

Is there any significance testing method that would help me properly assess the f-score difference between Alice and Bob?

Follow up: what would I do if I could have more than two possible labels for a data point, say $0$, $1$, or $2$?


I hope there is an analytic way of doing this, but I think a permutation test solves this for you.

Guess | Correct | Subject
  0        1         B
  1        1         A
 ...      ...       ...
  1        0         A

That's your data frame, containing the guessed value, the true value, and the subject making the guess.

To do a permutation test, you scramble the labels. Then you calculate the f-score for subject A and subject B, using the scrambled labels. Keep doing this over and over, keeping track of each subject's f-score. Finally, combine all of the permuted scors ratios with the true score ratio and test how unlikely a true ratio of whatever you calculated or more extreme is (typical hypothesis test).

The following R (pseudo) code should give you an idea about how to do the permutation test.

guess <- rbinom(1000,1,0.5) # just dummy data
correct <- rbinom(1000,1,0.5) # just dummy data
subject <- c(rep("A",500),rep("B",500))
f_score_ratio <- rep(NA,1000)
f_score_ratio[1] <- # Ratio of true f-scores
for (i in 2:1000){

    perm <- sample(subject,length(subject),replace=FALSE)
    guess_perm_A <- guess[perm=="A"]
    guess_perm_B <- guess[perm=="B"]
    # Here's where you calculate the f-score for each subject and 
    # and the ratio of one to the other (be consistent).

    f_score_ratio[i] <- # ratio of f-scores 

It doesn't really matter who is on top of the ratio as long as you're consistent.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.