What is a good measure (or set of measures) for the difference between two sample sets? I'm doing some software testing where we are measuring specific latencies.  Generally we run the same test several times to just eyeball the results and make sure that they are consistent across runs.  I usually do this by plotting the cumulative distributions for each run together on a graph, and look for an anomalies.
What more quantitative methods exist for doing this sort of thing?  I would ultimately like to be able to run my tests multiple times, and determine some sort of error bound, so I can say something like "I change variable X and the median time changed Y, but that is not significant".
Apologies for any vague handwaving above, I am not a statistician :)
 A: what kind of output are you getting? are you using regression? if you're trying to measure which set is best to use, you can use an F test using the sample errors, but I'm not sure what exactly you're taking about.
A: The general approach is to model your latencies in each condition with some distribution, like a Gaussian, and test whether the means of those two distributions are different, e.g. using a t-test. This lets you pick a significance threshold (p-value) and test for it automatically.
The Gaussian isn't a perfect model for this case, because it's symmetric, and you know that your latencies are the sum of the actual time to run your code in isolation, plus some random latency due to the OS swapping, waiting for I/O, ... in other words, you know that your code will never randomly run faster than the time to run it in isolation. But, it probably won't matter. I'd just use a t-test.
A: I would take your latency responses from running the operation and perform some tests for normality.  If these hold true, then you can create some confidence intervals using a standard normal table.  If your tests for normality don't hold true, then a t-table would be more appropriate.  You said that you normally plot your latency variables to find any anomalies.  How does your plot normally look?  Are these repeated operations being performed with the system being held constant?  Do you want the system held constant during your experiment?  Is the speed of the system being taken into account.  These are all things that could be considered to reduce your error in modeling of the time latency variables.  These things will provide for a better confidence interval.
