Calculating speedup confidence interval

Question

Summary

I'm trying to calculate the confidence interval for speedup of a program when I have many measurements for the old and new execution times.

Details

Speedup is calculated as: $\frac{time_{new}}{time_{old}}$, which is essentially a ratio. When looking for information about calculating the confidence interval of a ratio I looked at the question How to compute the confidence interval of the ratio of two normal means,which suggested using Fieller's theorem.

The Wikipedia article includes information about Approximate formulae and I believe my scenario would fall in case 1 because my standard error is relatively small compared to the measured execution time. So based on what Wikipedia says I should be able to use:

$$ Var \left(\frac{a}{b}\right) = \left(\frac{a}{b}\right)^2 \left(\frac{Var(a)}{a^2} + \frac{Var(b)}{b^2} \right) $$

This part makes sense to me, but where I got lost is when Wikipedia says:

From this a 95% confidence interval can be constructed in the usual way (degrees of freedom for $t^*$ is equal to the total number of values in the numerator and denominator minus 2).

I am familiar with using the steps described on Wikipedia to compute a confidence interval for a population, but I am not sure how to apply them when given this variance formula. The Wikipedia article on Fieller's theorem includes additional detail with an equation to use for "logged data" (I'm not certain what the means). But I don't think that equation is applicable for my problem.

Unfortunately I'm not able to make sense of how I would translate this to code (I would like implement my calculations in Ruby, but as long as I can figure out something like pseduocode for the equation the coding should not be a problem).

The question that I referenced earlier contains a like to an online calculator for computing the confidence interval of a quotient. I don't know how complex the code for this is, but if I could find something like it that would probably answer my question. I need to be able to calculate the CI in my own script rather than using an online calculator.

Answer 1

It sounds like the Wikipedia page is suggesting that

Q=["population ratio" - "estimate of ratio"]/["estimated sd of ratio estimate"]

should be approximately distributed as $t_{n-2}$ (where $n$ is the total number of observations). I'm not sure of the justification of that assertion, but taking it as given, forming a CI for the population ratio is straightforward.

Specifically, if I have understood the wikipedia page correctly, Q is a pivotal quantity (at least approximately) because its distribution doesn't change* as the parameters change (since it's always approximately $t_{n-2}$).

* or at least doesn't change much, if the approximation is a good one

You can use the $\alpha/2$ and $1-\alpha/2$ quantiles of a $t_{n-2}$ distribution as bounds on an interval for Q (i.e. if $Q$ is a random variable from a $t$ distribution, then $P(t_{\alpha/2})<Q<P(t_{1-\alpha/2}) = 1-\alpha$).

If you then replace $Q$ in that statement by that studentized ratio I started with and multiply through by the denominator and add the estimate of the ratio (which doesn't alter the inequality) then you will get an interval for the population ratio.

Answer 2

Assuming two Gaussian samples with equal variances, below is a R code calculating the Fieller confidence interval about the ratio of the means.

# data
y1 <- c(2, 2, 2, 3, 4, 4, 4)
y2 <- c(4, 5, 6, 7, 8)
# means
theta1 <- mean(y1)
theta2 <- mean(y2)
# residuals
residuals <- c(y1-mean(y1), y2-mean(y2))
# degrees of freedom
dof <- length(y1)+length(y2)-2
# estimated variance
sigma2 <- sum(residuals^2)/dof
# squared quantile
alpha <- 0.05
q2 <- qt(1-alpha/2, dof)^2
# discriminant 
f0 <- theta1^2 - q2*sigma2/length(y1)
f1 <- theta1*theta2 
f2 <- theta2^2 - q2*sigma2/length(y2)
D <- f1^2-f0*f2 
# lower bound (if D and f2 positive)
lwr <- (f1 - sqrt(D))/f2
# upper bound (if D and f2 positive)
upr <- (f1 + sqrt(D))/f2

Let's compare with the mratios package:

> options(digits=20)
> lwr; upr
[1] 0.31083095633020263
[1] 0.73533619300295594
> mratios::t.test.ratio(y1, y2, var.equal=TRUE)

    Ratio-t-test for equal variances

data:  x and y
t = -4.0504629365049123, df = 10, p-value = 0.0023218494780697131
alternative hypothesis: true ratio of means is not equal to 1
95 percent confidence interval:
 0.31083095633020263 0.73533619300295594

As mentioned in the comments of the code, this assumes $D \geq 0$ and $f_2 \geq 0$. There are two other possible situations:

• $D \geq 0$ and $f_2<0$ - in this case, with the same formulas for lwr and upr, the confidence region is $(-\infty, \texttt{upr}] \cup [\texttt{lwr}, +\infty)$ (complement of a finite interval);

• $D < 0$ and $f_2<0$ - in this case, the confidence region is the entire real line $(-\infty, +\infty)$.