0
$\begingroup$

I am trying to calculate the throughput percent increase between a software component A (baseline) and a newer version B, with 95% confidence level.

One option could be to calculate a 95% confidence interval for the difference between the two population means using Student's t-test distribution, $[c_l, c_h]$, using this formula

$$(\bar{x_a}-\bar{x_b}) \pm t_{0.025;df}\sqrt{s_{pooled}^2\left(\frac{1}{n_a}+\frac{1}{n_b}\right)}\qquad,df=n_a+n_b-2$$

and then we calculate the percent change confidence interval based on $\bar{x_a}$

$$\left[100\times\frac{c_l}{\bar{x_a}},100\times\frac{c_h}{\bar{x_a}}\right]$$

The issue, however, is that $\bar{x_a}$ is not a constant but a statistic, thus I don't think we can say the resulting percent change confidence interval is a 95% confidence interval (or can we?)

Another option could be to base the percent change confidence interval on the 95% confidence interval for A's mean, $[a_l, a_h]$. That is, we use $a_l$ and $ a_h$ to calculte the percent change:

$$\left[100\times\frac{c_l}{a_h},100\times\frac{c_h}{a_l}\right]$$

However, I feel this confidence interval is too conservative, and ultimately not a 95% confidence interval.

What would be the best way to calculate the percent change with a given confidence level?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.