I am trying to calculate the throughput percent increase between a software component A (baseline) and a newer version B, with 95% confidence level.

One option could be to calculate a 95% confidence interval for the difference between the two population means using Student's t-test distribution, $[c_l, c_h]$, using this formula

$$(\bar{x_a}-\bar{x_b}) \pm t_{0.025;df}\sqrt{s_{pooled}^2\left(\frac{1}{n_a}+\frac{1}{n_b}\right)}\qquad,df=n_a+n_b-2$$

and then we calculate the percent change confidence interval based on $\bar{x_a}$


The issue, however, is that $\bar{x_a}$ is not a constant but a statistic, thus I don't think we can say the resulting percent change confidence interval is a 95% confidence interval (or can we?)

Another option could be to base the percent change confidence interval on the 95% confidence interval for A's mean, $[a_l, a_h]$. That is, we use $a_l$ and $ a_h$ to calculte the percent change:


However, I feel this confidence interval is too conservative, and ultimately not a 95% confidence interval.

What would be the best way to calculate the percent change with a given confidence level?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.