Confidence interval of ratio estimator As an example, consider a program that executes on two computers, A and B.
Measuring the execution time of 3 executions each shows the following results:
System A: 10s, 10s, 4s
System B: 8s, 8s, 2s
With these values, we can calculate the mean and the confidence interval for both systems (considering Student's t distribution for a confidence of 90%): A: 8s +- 5.84
B: 6s +- 5.84
We can now calculate an average speedup between System A and B as 8s/6s = 
1.33.
My question is: Is it possible to calculate an error or confidence interval for this speedup? If yes, how?
A: You should take a look at the theory of 'ratio estimators' , there are references to find via google (e.g. http://www.math.montana.edu/~parker/PattersonStats/Ratio.pdf). The idea is that you can compute the mean and the variance of a ratio of random variables and then use the mean and variance to define confidence intervals. 
But I think you will need larger samples. 
@Matthias Diener
So for your example:
$y = (10, 10, 4), \bar{y}=8$, $x=(8,8,2), \bar{x}=6$
the ratio estimator for your speedup is $r=\frac{\bar{y}}{\bar{x}}=\frac{8}{6}=1.333$ and the variance of the estimator is $\sigma_r^2=\frac{1}{\bar{x}^2}\frac{s_r^2}{3}$, with $s_r^2=\frac{1}{3-1}\sum_{i=1}^3 (y_i - rx_i)^2$. 
The R-code looks like: 
y<-c(10, 10, 4)
x<-c(8, 8 , 2)

m.x<-mean(x)
m.y<-mean(y)

m.x
m.y

r<-m.y/m.x
r

s.r.2<-1/(length(x)-1) * sum(( y - r * x ) * ( y - r * x))

variance.r<- 1/m.x^2 * s.r.2/length(x)

stand.deviation.r<-sqrt(variance.r)

cat(paste("estimate for speedup:", round(r, digits=3), "with stand. deviation:", round(stand.deviation.r, digits=3)))

A: @f coopens gave the theoretical approach (+1), but here are a couple of other alternatives for those of us who like making the computer do the work rather than thinking hard:


*

*Bootstrapping, with such small sample sizes I would not trust the non-parametric bootstrap, but if you had big enough sample sizes then you could take bootstrap samples and for each pair of samples calculate the means and the ratio, then look at the quantiles of the bootstrapped ratios.

*Parametric bootstrapping, if you are happy with the normality/t assumptions then you can generate a bunch of "means" from a normal or t with the appropriate means and sd, then compute the ratios and look at the quantiles of the bootstrap ratios.

*Bayesian, (with small samples this will work best with an informative prior).  The nice thing with using McMC techniques to do Bayes is that you can look at the ratio of the means in each itteration and it will give you the posterior for the ratio without needing to work out the theory.
