First let me describe the situation I'm dealing with:
I'm looking at performance data for a software system. I have data for many versions of the software, including ongoing. For each version I have a series of values for the run-times of tests run against it, usually about 20 (of the same test). These generally look normally distributed (for each version).
What I want to know is, given some data for runs against a new version, is the new distribution different in a way that merits investigation. The comparison could be either against the previous version, or against a set of previous versions that have been selected as a having "stable" performance: a kind of baseline. Any kind of change could be relevant: a change in the mean, variance or shape could be significant.
Now, the different versions are, well, different, but for those in the baseline I think I can assume that they're effectively samples from the same distribution. So I've got $X_1, ... X_n, X'$, for $n \geq 1$, and I want to test whether $X'$ is relevantly similar to the $X_i$s in an automated fashion.
From what I've seen on the internet I've come up with a few options:
a) Kolmogorov-Smirnov test: either of $X'$ vs $X_n$, or of $X'$ vs $\bigcup X_i$.
b) T-test: similarly.
c) Mann-Whitney/Wilcoxon test?
Firstly, I'm not clear which would be better for my situation, as they both test for different kinds of "similarity", or whether I should use both and report some combination of the results.
Secondly, looking at the data, it looks like the $X_i$, while normally distributed, tend to move around a bit: so their means vary, but e.g. the variance is similar. If I just lump them into one big sample, then this information is lost; for example, it will look like the typical variance is much bigger than it actually is. For that reason I wondered whether I might be able to instead look at the distribution of the means and variances of the $X_i$s, and compare the mean and variance of $X'$ against that, somehow.
However, I'm unsure how to do that: I think the means should follow a t-distribution, so I ought to be able to estimate the probability of getting $\bar{X'}$ given that the sample came from the same distribution, but that's the wrong conditional probability! (although that's kind of just what a p-value is...) I can't do a proper t-test as that requires either more than one value to compare against, or the assumption that the variance is shared, which I'm not sure I have.
Finally, my inner Bayesian feels like I ought to be able to do better than producing p-values for rejecting the null hypotheses: surely I ought to be able to calculate a posterior probability that $X'$ is, say, drawn from the same distribution as the $X_i$s?
Apologies for the huge question; I hope this gives a reasonable idea of where I'm coming from! I'm mathematically trained, but I'm pretty unfamiliar with statistics, so I can cope with some maths.
Edit: I'm also familiar with R; I'm going to be using it to do the calculations... once I figure out what to calculate!