Return to Question

I am applying a two-sample t-test to determine whether we have software regressions on latency measurements.

Procedure

1. Run the test for build b1 and gather 60 latency measurements.
2. Run the test for build b2 and gather 60 latency measurements.
3. Calculate mean, stddev, for b1 and b2 separately.
4. Calculate the t-test score using this formula: $t = \frac{\bar{X_1} - \bar{X_2}}{\sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}}$
5. Interpret the score

I learned about this test from my engineering statistics textbook but I also saw it suggested in this question.

Not sure if it's relevant, but the variability should be relatively consistent since I'm using an RTOS system.

Problem

In one run of the test I calculated a score of -3.6 when I re-ran it, I got only -2.6.

Question

What does that tell me? Is the distribution not being captured by 60 samples? Is this test not appropriate here? Is -3.6 not a big difference to -2.6?

I thought the difference between a score of -3.6 and -2.6 is very large so I'm a little confused whether I'm approaching this the right way.

Clarification

1. When you say a significance of -3.6 and -2.6, are you referring to the t statistic or some other quantities?

I'm referring to the t statistic. The value resulting from this calculation:

$t = \frac{\bar{X_1} - \bar{X_2}}{\sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}}$

2. When you say you rerun the test, do you collect 60 extra latency time samples from your current build to compare with that of the previous build, or did you do something else?

In my procedure (edited) I repeated steps 2-5 but I did not use the old measurements obtained in step 2 from the previous test. The reason I did this was to see if my results would be consistent. I wanted to validate my approach.

I am confused by this question, because it seems to ask why you got two different results with two different datasets: surely that requires no explanation. How, then, am I misreading or misinterpreting it?

I believe I am taking two samples from the same distribution (for build b2) and that the results I get between them should be very comparable since I'm taking a sample size of 60. It seems to me that getting a result of -2.6 is significantly different than a score of -3.6 which correspond to confidence levels of (0.0046 and 0.00016 respectively). Isn't that an order of magnitude difference in confidence? Shouldn't I get repeatable results by re-running the test?

I am applying a two-sample t-test to determine whether we have software regressions on latency measurements.

Procedure

1. Run the test for build b1 and gather 60 latency measurements.
2. Run the test for build b2 and gather 60 latency measurements.
3. Calculate mean, stddev, for b1 and b2 separately.
4. Calculate the t-test score using this formula: $t = \frac{\bar{X_1} - \bar{X_2}}{\sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}}$
5. Interpret the score
6. Re-run the test for build b2 (call it b2*) and gather another 60 latency measurements (to check for consistency)
7. Calculate the test score for b2*.

I learned about this test from my engineering statistics textbook but I also saw it suggested in this question.

Not sure if it's relevant, but the variability should be relatively consistent since I'm using an RTOS system.

Problem

After collecting results for b1, I ran the experiment for b2 twice (b2 and b2*) so I could perform 2 separate two-sample-test calculations (b1/b2 and b1/b2*). The first dataset of b2 gave me a result of -3.6 (b1/b2). When I re-ran the test on b2, I re-calculated a two-sample-test score of -2.6 (b1/b2*). I was expecting the result to be more comparable. -2.6 seems like a big difference from -3.6.

Question

What does that tell me? Is the distribution not being captured by 60 samples? Is this test not appropriate here? Is -3.6 not a big difference to -2.6?

I thought the difference between a score of -3.6 and -2.6 is very large so I'm a little confused whether I'm approaching this the right way.

Clarification

1. When you say a significance of -3.6 and -2.6, are you referring to the t statistic or some other quantities?

I'm referring to the t statistic. The value resulting from this calculation:

$t = \frac{\bar{X_1} - \bar{X_2}}{\sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}}$

2. When you say you rerun the test, do you collect 60 extra latency time samples from your current build to compare with that of the previous build, or did you do something else?

In my procedure (edited) I repeated steps 2-5 but I did not use the old measurements obtained in step 2 from the previous test. The reason I did this was to see if my results would be consistent. I wanted to validate my approach.

I am confused by this question, because it seems to ask why you got two different results with two different datasets: surely that requires no explanation. How, then, am I misreading or misinterpreting it?

I believe I am taking two samples from the same distribution (for build b2) and that the results I get between them should be very comparable since I'm taking a sample size of 60. It seems to me that getting a result of -2.6 is significantly different than a score of -3.6 which correspond to confidence levels of (0.0046 and 0.00016 respectively). Isn't that an order of magnitude difference in confidence? Shouldn't I get repeatable results by re-running the test?

I am applying a two-sample t-test to determine whether we have software regressions on latency measurements.

Procedure

1. Run the test for build b1 and gather 60 latency measurements.
2. Run the test for build b2 and gather 60 latency measurements.
3. Calculate mean, stddev, for b1 and b2 separately.
4. Calculate the t-test score using this formula: $t = \frac{\bar{X_1} - \bar{X_2}}{\sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}}$
5. Interpret the score

I learned about this test from my engineering statistics textbook but I also saw it suggested in this question.

Not sure if it's relevant, but the variability should be relatively consistent since I'm using an RTOS system.

Problem

In one run of the test I calculated a score of -3.6 when I re-ran it, I got only -2.6.

Question

What does that tell me? Is the distribution not being captured by 60 samples? Is this test not appropriate here? Is -3.6 not a big difference to -2.6?

I thought the difference between a score of -3.6 and -2.6 is very large so I'm a little confused whether I'm approaching this the right way.

Clarification

1. When you say a significance of -3.6 and -2.6, are you referring to the t statistic or some other quantities?

I'm referring to the t statistic. The value resulting from this calculation:

$t = \frac{\bar{X_1} - \bar{X_2}}{\sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}}$

2. When you say you rerun the test, do you collect 60 extra latency time samples from your current build to compare with that of the previous build, or did you do something else?

In my procedure (edited) I repeated steps 2-5 but I did not use the old measurements obtained in step 2 from the previous test. The reason I did this was to see if my results would be consistent. I wanted to validate my approach.

I am applying a two-sample t-test to determine whether we have software regressions on latency measurements.

Procedure

1. Run the test for build b1 and gather 60 latency measurements.
2. Run the test for build b2 and gather 60 latency measurements.
3. Calculate mean, stddev, for b1 and b2 separately.
4. Calculate the t-test score using this formula: $t = \frac{\bar{X_1} - \bar{X_2}}{\sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}}$
5. Interpret the score

I learned about this test from my engineering statistics textbook but I also saw it suggested in this question.

Not sure if it's relevant, but the variability should be relatively consistent since I'm using an RTOS system.

Problem

In one run of the test I calculated a score of -3.6 when I re-ran it, I got only -2.6.

Question

What does that tell me? Is the distribution not being captured by 60 samples? Is this test not appropriate here? Is -3.6 not a big difference to -2.6?

I thought the difference between a score of -3.6 and -2.6 is very large so I'm a little confused whether I'm approaching this the right way.

Clarification

1. When you say a significance of -3.6 and -2.6, are you referring to the t statistic or some other quantities?

I'm referring to the t statistic. The value resulting from this calculation:

$t = \frac{\bar{X_1} - \bar{X_2}}{\sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}}$

2. When you say you rerun the test, do you collect 60 extra latency time samples from your current build to compare with that of the previous build, or did you do something else?

In my procedure (edited) I repeated steps 2-5 but I did not use the old measurements obtained in step 2 from the previous test. The reason I did this was to see if my results would be consistent. I wanted to validate my approach.

I am confused by this question, because it seems to ask why you got two different results with two different datasets: surely that requires no explanation. How, then, am I misreading or misinterpreting it?

I believe I am taking two samples from the same distribution (for build b2) and that the results I get between them should be very comparable since I'm taking a sample size of 60. It seems to me that getting a result of -2.6 is significantly different than a score of -3.6 which correspond to confidence levels of (0.0046 and 0.00016 respectively). Isn't that an order of magnitude difference in confidence? Shouldn't I get repeatable results by re-running the test?

I am applying a two-sample t-test to determine whether we have software regressions on latency measurements.

Procedure

• Run the test and gather 60 latency measurements.
• Calculate the two-sample t-test score against the previous build.
• Interpret the results (threshold on the t-test score and investigate further if needed)

I learned about this test from my engineering statistics textbook but I also saw it suggested in this question.

Not sure if it's relevant, but the variability should be relatively consistent since I'm using an RTOS system.

Problem

In one run of the test I calculated a significance of -3.6 when I re-ran it, I got only -2.6.

Question

What does that tell me? Is the distribution not being captured by 60 samples? Is this test not appropriate here? Is -3.6 not a big difference to -2.6?

I thought the difference between a score of -3.6 and -2.6 is very large so I'm a little confused whether I'm approaching this the right way.

I am applying a two-sample t-test to determine whether we have software regressions on latency measurements.

Procedure

1. Run the test for build b1 and gather 60 latency measurements.
2. Run the test for build b2 and gather 60 latency measurements.
3. Calculate mean, stddev, for b1 and b2 separately.
4. Calculate the t-test score using this formula: $t = \frac{\bar{X_1} - \bar{X_2}}{\sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}}$
5. Interpret the score

I learned about this test from my engineering statistics textbook but I also saw it suggested in this question.

Not sure if it's relevant, but the variability should be relatively consistent since I'm using an RTOS system.

Problem

In one run of the test I calculated a score of -3.6 when I re-ran it, I got only -2.6.

Question

What does that tell me? Is the distribution not being captured by 60 samples? Is this test not appropriate here? Is -3.6 not a big difference to -2.6?

I thought the difference between a score of -3.6 and -2.6 is very large so I'm a little confused whether I'm approaching this the right way.

Clarification

1. When you say a significance of -3.6 and -2.6, are you referring to the t statistic or some other quantities?

I'm referring to the t statistic. The value resulting from this calculation:

$t = \frac{\bar{X_1} - \bar{X_2}}{\sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}}$

2. When you say you rerun the test, do you collect 60 extra latency time samples from your current build to compare with that of the previous build, or did you do something else?

In my procedure (edited) I repeated steps 2-5 but I did not use the old measurements obtained in step 2 from the previous test. The reason I did this was to see if my results would be consistent. I wanted to validate my approach.