Comparing two sets of P values I have two sets of P-values. Each set contains about 1500 P-values that are computed by different versions -i.e. slightly different algorithms- of the same software. These two versions used the same input data and the P-values have the same meaning, but are slightly different. The algorithms test for an enrichment of certain cell types in a certain disease. Most cell types are not associated, leading to a negatively skewed distribution of P-values.
My goal is not to determine which algorithm performs the best, but I want to know whether there is a difference between the two sets of P-values. I am unsure whether it makes sense to compare two sets of P-values and if yes, how I would accomplish that. Any help would be greatly appreciated!
 A: This is a bit long for a comment. Let us start in the beginning. What reason do you want  p-values for? It may not be obvious that that question is even relevant, but it is. For example, the cost of an incorrect answer might be trivial or lethal in either direction but not both, false positive or false negative. So what the characteristics are of the p-value calculation should be ascertained in that context, the "What are you doing with those p-values?" question. Once that question is answered we can at least proceed to structure an answer. In a more neutral context, where the penalty of an incorrect answer is approximately balanced between false negative and false positive, one might anticipate  that the p-values that discriminate better would be preferable, that is, two tests could yield the same average probability but one of them could be useless and the other useful.
In either case, that of dissimilar or similar costs for an incorrect answer between false positive and false negative one can apply simulation studies with simulation truth data or one can collect structured truth data (where the correct answer is known), and then apply the two tests to determine which is working more appropriately for whatever the desired outcome is. I cannot think of any other way of approaching this problem, and if that is just my failure to imagine one, do let me know.
