How to compare a sample against some baseline data?

Question

First let me describe the situation I'm dealing with:

I'm looking at performance data for a software system. I have data for many versions of the software, including ongoing. For each version I have a series of values for the run-times of tests run against it, usually about 20 (of the same test). These generally look normally distributed (for each version).

What I want to know is, given some data for runs against a new version, is the new distribution different in a way that merits investigation. The comparison could be either against the previous version, or against a set of previous versions that have been selected as a having "stable" performance: a kind of baseline. Any kind of change could be relevant: a change in the mean, variance or shape could be significant.

Now, the different versions are, well, different, but for those in the baseline I think I can assume that they're effectively samples from the same distribution. So I've got $X_1, ... X_n, X'$, for $n \geq 1$, and I want to test whether $X'$ is relevantly similar to the $X_i$s in an automated fashion.

From what I've seen on the internet I've come up with a few options:

a) Kolmogorov-Smirnov test: either of $X'$ vs $X_n$, or of $X'$ vs $\bigcup X_i$.

b) T-test: similarly.

c) Mann-Whitney/Wilcoxon test?

Firstly, I'm not clear which would be better for my situation, as they both test for different kinds of "similarity", or whether I should use both and report some combination of the results.

Secondly, looking at the data, it looks like the $X_i$, while normally distributed, tend to move around a bit: so their means vary, but e.g. the variance is similar. If I just lump them into one big sample, then this information is lost; for example, it will look like the typical variance is much bigger than it actually is. For that reason I wondered whether I might be able to instead look at the distribution of the means and variances of the $X_i$s, and compare the mean and variance of $X'$ against that, somehow.

However, I'm unsure how to do that: I think the means should follow a t-distribution, so I ought to be able to estimate the probability of getting $\bar{X'}$ given that the sample came from the same distribution, but that's the wrong conditional probability! (although that's kind of just what a p-value is...) I can't do a proper t-test as that requires either more than one value to compare against, or the assumption that the variance is shared, which I'm not sure I have.

Finally, my inner Bayesian feels like I ought to be able to do better than producing p-values for rejecting the null hypotheses: surely I ought to be able to calculate a posterior probability that $X'$ is, say, drawn from the same distribution as the $X_i$s?

Apologies for the huge question; I hope this gives a reasonable idea of where I'm coming from! I'm mathematically trained, but I'm pretty unfamiliar with statistics, so I can cope with some maths.

Edit: I'm also familiar with R; I'm going to be using it to do the calculations... once I figure out what to calculate!

Answer 1

I normally have a Bayesian bias, but If I understand you correctly, I think Frequentist hypothesis testing is fairly appropriate here.

I think you probably want to do a t-test with of each $X_i$ with $X'$. This will tell you whether the mean of a given version is significantly different than the mean of the baseline.

You can then also do a F-test, which tests if the variances are different.

You may also consider computing Bayesian credible interval for the difference of means and seeing if 0 is outside that. Check out Bolstad's book Chapter 12. There's probably also a credible interval for the ratio of variances.

With more sophisticated Bayesian approaches you could probably sqeeze a tiny bit more information out of data, but I think you're probably much better off doing this and collecting more data if you need it.

I don't think you want to pool the $X_i$'s because that will tell you whether the sample weighted mean of version tests is different than $X'$. If I understand you, this is not what you're interested in, you're interested in which versions are different.

If you're interested in which versions go beyond the normal variation between versions, then you'll have to do something more complex.