Reporting results of random seeds and bootstrapping (stats) I am trying to compare my model's (a neural network) results with human-scored results on the same type of data. However, there are no ground truth labels (humans disagree on classifying these results), so Cohen's kappa is used to get the measure of agreement.
I hypothesize that my model is performing as well as humans at classifying this data, and this seems apparent when you eyeball boxplots of the two samples. However, I need a more rigorous test. My test sample (n=400) is not the same sample as the human results (n<100) that I'd like to compare against (also, I only have access to the kappa values for the human results, nothing else).
Given the smaller human sample, I believe that I should bootstrap both samples to try to approximate more closely the population statistics of the unseen data. But I'm not sure where to go from there. Should I be comparing the sample distributions? Should it be one or two-tailed (given that I'm hoping to perform as well or better)? Etc. I am not making any normality assumptions, so I guess a non-parametric test would be appropriate.
Additionally, and less important than the result above, I wanted to compute statistics on how much the random seed influences the results, so I ran the final model on 5 different seeds. The S.D. value of the median of the (5) sample kappas sets is tiny. But that then opened up another question: when I'm bootstrapping my results above, should I only bootstrap against 1 of these sets? And why?
 A: This is only a partial answer making a lot of assumptions which are compatible with the OP text but not explicitly stated by he/she. The usual path would be to post comments trying to tease out the details of the problem, but I need to make so many assumptions that I think it is more efficient to state them here and if they do not hold, the OP should just disregard the answer.
Assumptions
I assume:

*

*There is a unique test set T (the OP state that for this test set n< 100)


*there are k  humans that provided asnwers to the same test T. Each human $i$ response is $R_i$.


*there is no gold standard response $R^*$ to the test.


*The OP has a system that can run for T and computes the response $R_0$.


*The OP would lke to claim that his system has human level performance, or that  $R_0$ can be considered as good as the other $R_i$


*The OP can compute the kappa between any pair of human responses $\kappa_{ij}$ ($k(k-1)/2$ of such kappa)
1st solution
summary:  compute an approximation of the gold standadrd and measure accuracy of each solution against is.

*

*I believe there is literature of composing all human answers $R_i$ into a approximation of the gold standard $R^*$. (Many many years ago I was taught that Mycin - one of the first "expert systems" - was declared better than humans because it matched a gold standard approximation of the answers for problems for which the specialists had some disagreements).

The obvious solution is to stipulate that if a majority (50% plus 1 - actually I think this is called plurality in English)  of the humans answered $x$ for a particular data point in T, x is the correct answer. Of a data point has no majority answer it is removed from T. Or the OP may decide that a super majority is needed to declare a $x$ as the correct answer, for example 2/3 of the answers agree on $x$.

*

*with this constructed approximation to the gold standard $\tilde{R}$, compute the accuracy of each human and the system, and show where the accuracy of the system is placed in relation to the humans (is it better, in the top 10% and so on)


*there is no statistical test to be applied. The claim is based in a simple comparison of the system's accuracy against each human's accuracy.
2nd solution
summary: compute the kappa between the system and the humans, and show that this set of kappas is not much different than the set of kappas between all humans. This solution does not compute the $R^*$.

*

*compute the set of kappas between the system and each human $\kappa_{0i}$. The ser ok these k kappas is $\{\kappa_{0i}\}$


*the set of all pair human kappas is denoted as $\{\kappa_{ij}\}$.


*I believe that the claim that the system is similar to humans is the same as claiming that the distribution of $\{\kappa_{0i}\}$ is similar to the distribution $\{\kappa_{ij}\}$.


*there are statistical test that verify if two sets of data are from different distribution. I have no experience with these tests but two of them are Kolmogorov–Smirnov test and the Cramér–von_Mises_criterion. I cannot provide any advice on how to choose between these two.


*for both theses tests you want the two sample version.


*a big problem with these tests is that they can verify if the two samples are not or are unlikely from the same distribution. But the test do not show that the two samples came from the same distribution. That is the null hypothesis of the tests is that they come from the same distribution, and a low p-value would indicate that the null hypothesis is not true. But there is no way in the test to show that the null hypothesis is true.


*you may, at your own risk, follow the common error that if the p-value is > 0.05 that you claim that the two sets came from the same distribution - that may or not be acceptable in your context. If you make that claim, that mean that the set of kappas between the system and the humans is not different than the set of kappas among humans, which is our approximation to the claim that the system is as good as humans.
