How to assess the effect of optimisation on software performance improvement based on five trials with and without optimisation? [This question is similar to How to assess the improvement from a treatment?]
I'm attempting to characterize what appears to be an obvious improvement in performance as a result of some changes to a software application. I've run the application five times without the optimization ('A' represents the timing for such) and five times with the optimization ('B').
A = [12.6, 12.6, 12.5, 12.7, 12.7]
B = [4.3, 3.3, 4.3, 3.2, 3.3]
As a layman, this appears to represent a reasonable improvement in performance. However, I don't know how to characterize the legitimacy of that improvement.
Is my sample size large enough to allow me to claim anything from a statistical point of view?
Can I calculate a confidence with any statement of performance improvement or the measurements themselves?
 A: Here is a link to some NIST pages:


*

*http://www.itl.nist.gov/div898/handbook/ppc/ppc.htm

*http://www.itl.nist.gov/div898/handbook/ppc/section5/ppc52.htm

*http://www.itl.nist.gov/div898/handbook/prc/prc.htm
Here is the one that I think is most interesting to you:


*

*http://www.itl.nist.gov/div898/handbook/prc/section2/prc2.htm
Bottom lines:


*

*Mileage is going to vary based on system, measurements and changes.

*A good starting point is to look at the change in mean and change in standard deviation.  Make sure that you get enough samples to make an adequate comparison.

A: Independent groups t-test
The standard approach to data like this is to perform an independent groups t-test.
You can test this in R using the following simple command.
t.test(c(12.6, 12.6, 12.5, 12.7, 12.7), c(4.3, 3.3, 4.3, 3.2, 3.3))

The results are:
    Welch Two Sample t-test

data:  c(12.6, 12.6, 12.5, 12.7, 12.7) and c(4.3, 3.3, 4.3, 3.2, 3.3)
t = 34.8517, df = 4.174, p-value = 2.638e-06
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 8.239347 9.640653
sample estimates:
mean of x mean of y 
    12.62      3.68 

P is much less than .05, the 95% confidence intervals of the difference is much greater than zero, so all else being equal you would probably conclude that there is a real difference.
Broader considerations
I imagine that software optimisation involves a wide range of other issues that you would have to consider beyond standard statistical significance. I'd be asking question like


*

*Were the tests in the two conditions run under otherwise identical conditions?

*Were the tests run under a sufficiently broad range of conditions that the observed changes would generalise to the conditions where the software is typically run?

