# Hypothesis testing - speeds before and after

I have the following data:

   A   |   B
---------------
9,794  |  10,7098
9,8022 |  10,5176
9,7055 |  11,1039
9,7091 |  11,1474
10,1882|  10,4693
10,2204|  10,8072
9,8221 |  11,2713
9,9272 |  11,2888
10,0855|  10,9026
10,108 |  10,872
10,1433|  10,9649
10,1432|  10,9805
9,7451 |  10,5126
9,8088 |  10,4722
10,0242|  10,6095
10,0402|  10,7719
9,5203 |  11,0741
9,5211 |  11,0357
9,4078 |  11,1146
9,4367 |  11,1176
10,3585|  11,0609
10,3234|  11,0389
10,0555|  10,6333
9,9836 |  10,4394
9,8306 |  10,2537
9,8836 |  10,3016
9,9489 |  10,1945
9,9176 |  10,339
10,0038|  10,639
9,9711 |  10,6681
-----------------
9,91431|  10,77706 - Average
0,2404 |  0,31310438 - Standard deviation


It represents the seconds a computer program did when using configuration A and B. I run the program 30 times with configuration A, and another 30 times with configuration B.

I want to construct a hypothesis test that proves configuration B runs slower than A (or configuration B runs with a different time than A).

What test should I use and what values go where?

I did some tests using the formulas and got this value por z (or t, dont remember): -11,769. Isn't this number really low? The critical point for 5% is 1.96 (two-tailed) and -11.769 is really deep into the critical zone (rejecting H0) with high probability. I think i'm doing something wrong so please help me. Thanks!

• What makes you think your test results might be misleading you? Have you graphed the data? (The graph will make the conclusions obvious.)
– whuber
Commented Nov 20, 2014 at 16:31
• draw a box plot with both variables, they are clearly very different appart from few very small values. It looks normal to have a very significant p-value Commented Nov 20, 2014 at 16:55
• Thanks guys! I've never done hypothesis testing and I thought the values I was getting were wrong but yes I can see the values are different. Commented Nov 20, 2014 at 16:58
• Just as a quick clarification - it looks like you are using the comma as the indication that separates the integer part of the number from the fractional part of the number. Am I reading that correctly? Commented Nov 20, 2014 at 20:12
• @DonDresserLatentView, exactly. The number is expressed in seconds using comma as a decimal separator Commented Nov 20, 2014 at 23:00

I get a similar value for the t-test of difference in means between the two samples, and a t test is an appropriate test for this question.

There are some fussy technical points about exactly how you figure the standard error for the difference of means, that I am not sure I can answer correctly (although I am pretty sure that for any of them, you would want to start by calculating the estimate of the standard deviation from a sample, which is slightly larger than the standard deviation you appear to have calculated in your data display - I think you used the formula for calculating the standard deviation of a population there) - but any of the variations still produce t statistics of well over 11 - which is way out in the tail of the t distribution, so you would certainly reject a null hypothesis that the means were the same.

Since I already collected the data and draw some things, I wil post them here too.

First is a density plot of all values

Second is a box plot.

• The comma was used as a decimal point. Thus, you should have multiplied all those small outliers by $10$ to make them comparable to the others.
– whuber
Commented Nov 20, 2014 at 23:50

It seems to me that usually with timings you'd be expecting (and interested in detecting) scale shifts. This would corresponding to looking at the percentage faster or slower.

As such, a location-shift in log-time would probably be of prime interest.

I don't think there's likely to be much of a problem with a t-test on the logs, but a Wlcoxon-Mann-Whitney should work very well (the statistic and the p-value will be unaffected by the log-transform, but it's better to work on the log scale because you can get a nice CI for the location-shift in the logs, which when exponentiated will be a CI for the scale shift).

(It's not completely clear whether you're after a one-sided test or a two-sided test; I've done this as if you're interested in the two-sided alternative.)

wilcox.test(x=log(AB$B),y=log(AB$A),exact=TRUE,conf.int=TRUE)

Wilcoxon rank sum test

data:  log(AB$B) and log(AB$A)
W = 892, p-value = 1.133e-15
alternative hypothesis: true location shift is not equal to 0
95 percent confidence interval:
0.06819419 0.09942080
sample estimates:
difference in location
0.0842329


and the CI for the scale shift:

> exp(wilcox.test(x=log(AB$B),y=log(AB$A),exact=TRUE,conf.int=TRUE)$conf.int) [1] 1.070573 1.104531 attr(,"conf.level") [1] 0.95  So we might say that$B$is clearly slower than$A$, and runs somewhere in the region of 7% to 10.5% slower. An equal-variance t-test on the logs produces a similar (but slightly narrower) interval for the percentage by which$B\$ is slower.

Either would seem to be eminently suitable as a way of looking at the data.