How can I determine if there's a statistically significant difference between two averages?

Question

I'm writing a benchmark program in C# and Java as a first assignment for a CS class. We're supposed to write some kind of report about our methods and results from the benchmark, and I'd like to add a statistical component to mine.

I've got 15 run time samples from each language, and the mean and standard deviation from each. How can I determine whether there is a statistically significant difference between them?

Answer 1

If your run time samples for each language are roughly normally distributed* (which is likely the case), then you could use a t-test, in particular, an independent two-sample t-test with unequal variances.

If you have R installed, you could do this by running t.test(x = c_sharp_samples, y = java_samples).

If, however, you want to run the test by hand, first calculate:

• $t = \frac{\bar{X_1} - \bar{X_2}}{s_{\bar{X_1} - \bar{X_2}}}$, where $s_{\bar{X_1} - \bar{X_2}} = \sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}$ and $\bar{X_1}$ is the sample mean of the C# samples, $s_1$ is the sample standard deviation of the C# samples, $n_1$ is the number of C# samples, and so on.
• $df = \frac{(s_1^2 / n_1 + s_2^2 / n_2)^2}{(s_1^2 / n_1)^2 / (n_1 - 1) + (s_2^2 / n_2)^2 / (n_2 - 1)}$.

Then $t$ (approximately) follows a Student's t distribution with $df$ degrees of freedom, so lookup $t$ in the appropriate table (or using some t distribution calculator).

*Even if your run time samples for each language aren't normally distributed, 15 samples is probably enough for a normal approximation (i.e., the CLT) to kick in, so you should be fine. But if you want to be formal about it and don't want to make this normal assumption, you could use the (non-parametric) Mann Whitney Test instead.

Answer 2

A permutation test is another possibility, although I think for the problem you describe the alternatives that have been mentioned will be superior.

Answer 3

It sounds like what you want is to use a t-test (here is the wikipedia page).

If you do not assume your observations distribute normally then try this, the Mann-Whitney U test, (but it can not be computed from mean/sd alone).

Make sure that your observations are independent, so that the validity of the t-test is preserved.

Good luck, and read more about this before you do things!