First let me describe the situation I'm dealing with:

I'm looking at performance data for a software system. I have data for many versions of the software, including ongoing. For each version I have a series of values for the run-times of tests run against it, usually about 20 (of the same test). These generally look normally distributed (for each version).

What I want to know is, given some data for runs against a new version, is the new distribution different in a way that merits investigation. The comparison could be either against the previous version, or against a set of previous versions that have been selected as a having "stable" performance: a kind of baseline. Any kind of change could be relevant: a change in the mean, variance or shape could be significant.

Now, the different versions are, well, different, but for those in the baseline I think I can assume that they're effectively samples from the same distribution. So I've got $X_1, ... X_n, X'$, for $n \geq 1$, and I want to test whether $X'$ is relevantly similar to the $X_i$s in an automated fashion.

From what I've seen on the internet I've come up with a few options:

a) Kolmogorov-Smirnov test: either of $X'$ vs $X_n$, or of $X'$ vs $\bigcup X_i$.

b) T-test: similarly.

c) Mann-Whitney/Wilcoxon test?

Firstly, I'm not clear which would be better for my situation, as they both test for different kinds of "similarity", or whether I should use both and report some combination of the results.

Secondly, looking at the data, it looks like the $X_i$, while normally distributed, tend to move around a bit: so their means vary, but e.g. the variance is similar. If I just lump them into one big sample, then this information is lost; for example, it will look like the typical variance is much bigger than it actually is. For that reason I wondered whether I might be able to instead look at the distribution of the means and variances of the $X_i$s, and compare the mean and variance of $X'$ against that, somehow.

However, I'm unsure how to do that: I think the means should follow a t-distribution, so I ought to be able to estimate the probability of getting $\bar{X'}$ given that the sample came from the same distribution, but that's the wrong conditional probability! (although that's kind of just what a p-value is...) I can't do a proper t-test as that requires either more than one value to compare against, or the assumption that the variance is shared, which I'm not sure I have.

Finally, my inner Bayesian feels like I ought to be able to do better than producing p-values for rejecting the null hypotheses: surely I ought to be able to calculate a posterior probability that $X'$ is, say, drawn from the same distribution as the $X_i$s?

Apologies for the huge question; I hope this gives a reasonable idea of where I'm coming from! I'm mathematically trained, but I'm pretty unfamiliar with statistics, so I can cope with some maths.

Edit: I'm also familiar with R; I'm going to be using it to do the calculations... once I figure out what to calculate!

  • $\begingroup$ Are you still interested in this question? I think I might have some suggestions. $\endgroup$ Feb 4 '13 at 23:23
  • $\begingroup$ Absolutely! That would be very helpful. $\endgroup$ Feb 6 '13 at 2:45
  • $\begingroup$ To clarify, there is only one kind of test that you are considering here, right? The $X_i$'s are vectors with the test values for version $i$ in them, right? $\endgroup$ Feb 6 '13 at 22:56
  • $\begingroup$ Also, there is sort of normal variation between versions, which you don't want to detect? You just want to detect variation that's beyond the normal version to version variance? $\endgroup$ Feb 6 '13 at 23:01
  • $\begingroup$ The Xis are vectors with the test values for version i, yes. Can't say much about them; that's kind of the problem! Part of the problem is that I'm not sure what I'm looking for: the KS test seems to do a decent job of picking out things that look "different" so far. I'm looking for a difference in shape, or in location, basically. $\endgroup$ Feb 8 '13 at 3:01

I normally have a Bayesian bias, but If I understand you correctly, I think Frequentist hypothesis testing is fairly appropriate here.

I think you probably want to do a t-test with of each $X_i$ with $X'$. This will tell you whether the mean of a given version is significantly different than the mean of the baseline.

You can then also do a F-test, which tests if the variances are different.

You may also consider computing Bayesian credible interval for the difference of means and seeing if 0 is outside that. Check out Bolstad's book Chapter 12. There's probably also a credible interval for the ratio of variances.

With more sophisticated Bayesian approaches you could probably sqeeze a tiny bit more information out of data, but I think you're probably much better off doing this and collecting more data if you need it.

I don't think you want to pool the $X_i$'s because that will tell you whether the sample weighted mean of version tests is different than $X'$. If I understand you, this is not what you're interested in, you're interested in which versions are different.

If you're interested in which versions go beyond the normal variation between versions, then you'll have to do something more complex.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.