I am applying a two-sample t-test to determine whether we have software regressions on latency measurements.

Procedure

1. Run the test for build b1 and gather 60 latency measurements.
2. Run the test for build b2 and gather 60 latency measurements.
3. Calculate mean, stddev, for b1 and b2 separately.
4. Calculate the t-test score using this formula: $t = \frac{\bar{X_1} - \bar{X_2}}{\sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}}$
5. Interpret the score

I learned about this test from my engineering statistics textbook but I also saw it suggested in this question.

Not sure if it's relevant, but the variability should be relatively consistent since I'm using an RTOS system.

Problem

In one run of the test I calculated a score of -3.6 when I re-ran it, I got only -2.6.

Question

What does that tell me? Is the distribution not being captured by 60 samples? Is this test not appropriate here? Is -3.6 not a big difference to -2.6?

I thought the difference between a score of -3.6 and -2.6 is very large so I'm a little confused whether I'm approaching this the right way.

Clarification

1. When you say a significance of -3.6 and -2.6, are you referring to the t statistic or some other quantities?

I'm referring to the t statistic. The value resulting from this calculation:

$t = \frac{\bar{X_1} - \bar{X_2}}{\sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}}$

2. When you say you rerun the test, do you collect 60 extra latency time samples from your current build to compare with that of the previous build, or did you do something else?

In my procedure (edited) I repeated steps 2-5 but I did not use the old measurements obtained in step 2 from the previous test. The reason I did this was to see if my results would be consistent. I wanted to validate my approach.