Why is the two-sample test giving me inconsistent results?

Question

I am applying a two-sample t-test to determine whether we have software regressions on latency measurements.

Procedure

1. Run the test for build b1 and gather 60 latency measurements.
2. Run the test for build b2 and gather 60 latency measurements.
3. Calculate mean, stddev, for b1 and b2 separately.
4. Calculate the t-test score using this formula: $t = \frac{\bar{X_1} - \bar{X_2}}{\sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}}$
5. Interpret the score
6. Re-run the test for build b2 (call it b2*) and gather another 60 latency measurements (to check for consistency)
7. Calculate the test score for b2*.

I learned about this test from my engineering statistics textbook but I also saw it suggested in this question.

Not sure if it's relevant, but the variability should be relatively consistent since I'm using an RTOS system.

Problem

After collecting results for b1, I ran the experiment for b2 twice (b2 and b2*) so I could perform 2 separate two-sample-test calculations (b1/b2 and b1/b2*). The first dataset of b2 gave me a result of -3.6 (b1/b2). When I re-ran the test on b2, I re-calculated a two-sample-test score of -2.6 (b1/b2*). I was expecting the result to be more comparable. -2.6 seems like a big difference from -3.6.

Question

What does that tell me? Is the distribution not being captured by 60 samples? Is this test not appropriate here? Is -3.6 not a big difference to -2.6?

I thought the difference between a score of -3.6 and -2.6 is very large so I'm a little confused whether I'm approaching this the right way.

Clarification

1. When you say a significance of -3.6 and -2.6, are you referring to the t statistic or some other quantities?

I'm referring to the t statistic. The value resulting from this calculation:

$t = \frac{\bar{X_1} - \bar{X_2}}{\sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}}$

2. When you say you rerun the test, do you collect 60 extra latency time samples from your current build to compare with that of the previous build, or did you do something else?

In my procedure (edited) I repeated steps 2-5 but I did not use the old measurements obtained in step 2 from the previous test. The reason I did this was to see if my results would be consistent. I wanted to validate my approach.

I am confused by this question, because it seems to ask why you got two different results with two different datasets: surely that requires no explanation. How, then, am I misreading or misinterpreting it?

I believe I am taking two samples from the same distribution (for build b2) and that the results I get between them should be very comparable since I'm taking a sample size of 60. It seems to me that getting a result of -2.6 is significantly different than a score of -3.6 which correspond to confidence levels of (0.0046 and 0.00016 respectively). Isn't that an order of magnitude difference in confidence? Shouldn't I get repeatable results by re-running the test?

Answer 1

The t-statistic for each repetition of the test under Build b2 ought to follow, approximately, a non-central t-distribution with a little less than $2\times(60-1)=118$ degrees of freedom (you suggest the variances should be roughly equal); & this will be well approximated by a Gaussian distribution with unit variance. So if the non-centrality parameters are equal, their difference ought to follow, again approximately, a Gaussian distribution with zero mean and a variance of two, & the magnitude of the difference will be greater than one (the difference you observed) with a probability of nearly one half. There's nothing surprising about your results (though it'd still be sensible to look at plots of the latency times).

Your intuition is perhaps at fault; you write:—

I believe I am taking two samples from the same distribution (for build b2) and that the results I get between them should be very comparable since I'm taking a sample size of 60.

The results—the differences in sample means from the sample mean for Build b1—ought indeed to be comparable if the Build b2 samples are from the same distribution, & will tend to get closer together should you increase the sample size (the variances of the sample means decrease); the t-statistics, on the other hand, are already scaled by estimates of the variances of the sample means, so won't tend to get closer together.

---

If it's a concern that conditions might vary between batches of latency measurements (different times of day, interference from other running processes, &c.) consider a mixed-model.

Assessment of whether your sample sizes provide powerful-enough tests or narrow-enough confidence intervals is the domain of power-analysis.