2
$\begingroup$

Summary

I'm trying to calculate the confidence interval for speedup of a program when I have many measurements for the old and new execution times.

Details

Speedup is calculated as: $\frac{time_{new}}{time_{old}}$, which is essentially a ratio. When looking for information about calculating the confidence interval of a ratio I looked at the question How to compute the confidence interval of the ratio of two normal means,which suggested using Fieller's theorem.

The Wikipedia article includes information about Approximate formulae and I believe my scenario would fall in case 1 because my standard error is relatively small compared to the measured execution time. So based on what Wikipedia says I should be able to use:

$$ Var \left(\frac{a}{b}\right) = \left(\frac{a}{b}\right)^2 \left(\frac{Var(a)}{a^2} + \frac{Var(b)}{b^2} \right) $$

This part makes sense to me, but where I got lost is when Wikipedia says:

From this a 95% confidence interval can be constructed in the usual way (degrees of freedom for $t^*$ is equal to the total number of values in the numerator and denominator minus 2).

I am familiar with using the steps described on Wikipedia to compute a confidence interval for a population, but I am not sure how to apply them when given this variance formula. The Wikipedia article on Fieller's theorem includes additional detail with an equation to use for "logged data" (I'm not certain what the means). But I don't think that equation is applicable for my problem.

Unfortunately I'm not able to make sense of how I would translate this to code (I would like implement my calculations in Ruby, but as long as I can figure out something like pseduocode for the equation the coding should not be a problem).

The question that I referenced earlier contains a like to an online calculator for computing the confidence interval of a quotient. I don't know how complex the code for this is, but if I could find something like it that would probably answer my question. I need to be able to calculate the CI in my own script rather than using an online calculator.

Improve this question
$\endgroup$
10
  • $\begingroup$ You don't translate the quoted material into code! Simply skip down to the next two lines in the Wikipedia article which do give the formulas to use. $\endgroup$ – whuber♦ May 11 '16 at 20:28
  • $\begingroup$ @whuber do you know what the article means when it says "logged data is used"? And why the formula you mention have logorithms in them? Do they still work if I'm not using logorithms? I don't think that makes sense for my use case, but I'm not sure. $\endgroup$ – Gabriel Southern May 11 '16 at 20:34
  • $\begingroup$ Because "logged data" is a (terribly informal) way of saying "if the data are logarithms of the quantities you are taking the ratios of." If you're not taking logarithms, then why do you care at all about this quotation and what follows it? $\endgroup$ – whuber♦ May 11 '16 at 20:35
  • 1
    $\begingroup$ First and foremost, this site encourages you to improve your question. It discourages extended discussion because you are expected to be able to improve the question without needing to engage in a prolonged interchange in order to decide what needs editing. $\endgroup$ – whuber♦ May 11 '16 at 22:03
  • 1
    $\begingroup$ I removed the extraneous equations related to "logged data" since it seems like they are not applicable to me, and I included a link to the article on using confidence interval that I referenced in the comment. Hopefully that makes the question clearer. $\endgroup$ – Gabriel Southern May 11 '16 at 22:13
2
$\begingroup$

It sounds like the Wikipedia page is suggesting that

Q=["population ratio" - "estimate of ratio"]/["estimated sd of ratio estimate"]

should be approximately distributed as $t_{n-2}$ (where $n$ is the total number of observations). I'm not sure of the justification of that assertion, but taking it as given, forming a CI for the population ratio is straightforward.

Specifically, if I have understood the wikipedia page correctly, Q is a pivotal quantity (at least approximately) because its distribution doesn't change* as the parameters change (since it's always approximately $t_{n-2}$).

* or at least doesn't change much, if the approximation is a good one

You can use the $\alpha/2$ and $1-\alpha/2$ quantiles of a $t_{n-2}$ distribution as bounds on an interval for Q (i.e. if $Q$ is a random variable from a $t$ distribution, then $P(t_{\alpha/2})<Q<P(t_{1-\alpha/2}) = 1-\alpha$).

If you then replace $Q$ in that statement by that studentized ratio I started with and multiply through by the denominator and add the estimate of the ratio (which doesn't alter the inequality) then you will get an interval for the population ratio.

Improve this answer
$\endgroup$
2
  • $\begingroup$ can I ask what $t_{n-2}$ is? $\endgroup$ – Gabriel Southern May 12 '16 at 22:00
  • $\begingroup$ A t distribution with $n-2$ degrees of freedom as it describes in the page you link to -- I overload the use of the subscript by later using it (e.g. in $t_{\alpha/2}$) to represent a quantile of the $t_{n-2}$ distribution but I hoped that shift would be sufficiently clear in context. $\endgroup$ – Glen_b May 13 '16 at 1:12
1
$\begingroup$

Assuming two Gaussian samples with equal variances, below is a R code calculating the Fieller confidence interval about the ratio of the means.

# data
y1 <- c(2, 2, 2, 3, 4, 4, 4)
y2 <- c(4, 5, 6, 7, 8)
# means
theta1 <- mean(y1)
theta2 <- mean(y2)
# residuals
residuals <- c(y1-mean(y1), y2-mean(y2))
# degrees of freedom
dof <- length(y1)+length(y2)-2
# estimated variance
sigma2 <- sum(residuals^2)/dof
# squared quantile
alpha <- 0.05
q2 <- qt(1-alpha/2, dof)^2
# discriminant 
f0 <- theta1^2 - q2*sigma2/length(y1)
f1 <- theta1*theta2 
f2 <- theta2^2 - q2*sigma2/length(y2)
D <- f1^2-f0*f2 
# lower bound (if D and f2 positive)
lwr <- (f1 - sqrt(D))/f2
# upper bound (if D and f2 positive)
upr <- (f1 + sqrt(D))/f2

Let's compare with the mratios package:

> options(digits=20)
> lwr; upr
[1] 0.31083095633020263
[1] 0.73533619300295594
> mratios::t.test.ratio(y1, y2, var.equal=TRUE)

    Ratio-t-test for equal variances

data:  x and y
t = -4.0504629365049123, df = 10, p-value = 0.0023218494780697131
alternative hypothesis: true ratio of means is not equal to 1
95 percent confidence interval:
 0.31083095633020263 0.73533619300295594

As mentioned in the comments of the code, this assumes $D \geq 0$ and $f_2 \geq 0$. There are two other possible situations:

  • $D \geq 0$ and $f_2<0$ - in this case, with the same formulas for lwr and upr, the confidence region is $(-\infty, \texttt{upr}] \cup [\texttt{lwr}, +\infty)$ (complement of a finite interval);

  • $D < 0$ and $f_2<0$ - in this case, the confidence region is the entire real line $(-\infty, +\infty)$.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.