I'm trying to compare several methods by their performance on a set of synthetic data samples. For each method, I obtain a performance value between 0 and 1 for each of those samples. Then I plot a graph with the average performance per method
The problem now is that the achievable quality per sample varies strongly between different samples (if you wonder why, I generate random graphs and evaluate community detection methods on them, and sometimes strange things happen, like elements from the same community get disconnected due to sparseness etc). So showing error bars based on standard deviation or standard error tend to get really large.
Imagine one method yields [1, 0.5, 1], and the other one (one the same three samples) [0.5, 0.25, 0.5]. Which measure can I apply to *de*emphasize the between-sample variance in the series and emphasize the fact that method 1 always outperforms method 2? Or, to put it differently, how can I test whether method 1 is significantly better than method 2 without being mislead by the different range of the indiviual datapoints? (Also note that I typically have more than two methods to compare, this is just for the example)
Update One thing I've done is to count, for each method, how many times its performance is within 95% of the top performance. The picture pretty much speaks in favor of sample-based variance, not robust vs less robust methods. However, I'm still uncertain about how to generate a statistically valid statement from that..?
Update two years after Just found this answer again. Simply for anyone who stumbles across this: I went with a sign-test: How many times is method x better than method y. Then the null-hypothesis is that one should be better than the other 50% of the time if there's no difference - computing the probability that the actual number of wins/losses stems from a 0.5 coinflip can be computed via the binomial distribution, and serves as your $p$.