Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm writing a benchmark program in C# and Java as a first assignment for a CS class. We're supposed to write some kind of report about our methods and results from the benchmark, and I'd like to add a statistical component to mine.

I've got 15 run time samples from each language, and the mean and standard deviation from each. How can I determine whether there is a statistically significant difference between them?

share|improve this question
up vote 8 down vote accepted

If your run time samples for each language are roughly normally distributed* (which is likely the case), then you could use a t-test, in particular, an independent two-sample t-test with unequal variances.

If you have R installed, you could do this by running t.test(x = c_sharp_samples, y = java_samples).

If, however, you want to run the test by hand, first calculate:

  • $t = \frac{\bar{X_1} - \bar{X_2}}{s_{\bar{X_1} - \bar{X_2}}}$, where $s_{\bar{X_1} - \bar{X_2}} = \sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}$ and $\bar{X_1}$ is the sample mean of the C# samples, $s_1$ is the sample standard deviation of the C# samples, $n_1$ is the number of C# samples, and so on.
  • $df = \frac{(s_1^2 / n_1 + s_2^2 / n_2)^2}{(s_1^2 / n_1)^2 / (n_1 - 1) + (s_2^2 / n_2)^2 / (n_2 - 1)}$.

Then $t$ (approximately) follows a Student's t distribution with $df$ degrees of freedom, so lookup $t$ in the appropriate table (or using some t distribution calculator).

*Even if your run time samples for each language aren't normally distributed, 15 samples is probably enough for a normal approximation (i.e., the CLT) to kick in, so you should be fine. But if you want to be formal about it and don't want to make this normal assumption, you could use the (non-parametric) Mann Whitney Test instead.

share|improve this answer
This is perfect, thanks! – Carson Myers Sep 25 '11 at 17:06

A permutation test is another possibility, although I think for the problem you describe the alternatives that have been mentioned will be superior.

share|improve this answer

It sounds like what you want is to use a t-test (here is the wikipedia page).

If you do not assume your observations distribute normally then try this, the Mann-Whitney U test, (but it can not be computed from mean/sd alone).

Make sure that your observations are independent, so that the validity of the t-test is preserved.

Good luck, and read more about this before you do things!

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.