I have taken $N$ measurements $x_i$ of the time taken by a computer program to run a particular calculation, and would like to calculate a 95% confidence interval for the mean runtime. We can calculate the sample mean and standard deviation in the usual way:

$$ \bar{x} = \frac{1}{N} \left( \sum_{i=1}^N {x_i} \right); s = \sqrt{\frac{\sum_{i=1}^N (x_i - \overline{x})^2}{N-1} } $$

Now, as I understand it, if $N \ge 30$, we can appeal to the Central Limit Theorem and say that our sample mean $\bar{x}$ has come from a normal distribution with mean $\mu$ and standard deviation $\sigma / \sqrt{N}$, where $\mu$ and $\sigma$ are the population mean and standard deviation respectively. A 95% confidence interval for the mean runtime is therefore bounded by, for some $k_1, k_2$:

$$ \bar{x} \pm k_1 \frac{\sigma}{\sqrt{N}}\ or\ \bar{x} \pm k_2 \frac{s}{\sqrt{N}} $$

We can take $k_1$ from the normal distribution: $k_1 = Z_{0.975} = 1.96$. However, if we need to calculate a confidence interval without knowing $\sigma$, what value should we use for $k_2$?

  1. According to the paper Statistically Rigorous Java Performance Evaluation, we can also take $k_2$ from the normal distribution: $k_2 = Z_{0.975} = 1.96$. However, my understanding is that we should instead use Student's t-distribution with $N - 1$ degrees of freedom: for $N = 30$ this gives $k_2 = t_{(0.975, 29)} = 2.045$. Which of these is correct?

  2. The same paper claims that the above method (using the t-distribution) is valid even for $N < 30$. Is this true, given that we cannot apply the Central Limit Theorem in this case? If not, is there anything we can say about a confidence interval for the population mean in the case $N < 30$?

  • 1
    $\begingroup$ Often computer running times to do a specific task are far from normally distributed. So my answer provides information on confidence intervals (CIs) based on non-normal data. // Whenever the population variance is unknown, a CI for normal data is inherently a t CI (not a z CI) regardless of sample size. For large $n \ge 30$ (thus large degrees of freedom for t), and for a 95% CI only, you can use 1.96 instead of a cut-off value from a t distribution as an approximation. $\endgroup$ – BruceET Jul 22 '18 at 19:22
  • $\begingroup$ @BruceET - "Often computer running times to do a specific task are far from normally distributed." I am hopeful that when taking measurements, I can isolate my program from external system events that can cause outliers and/or a significant positive skew. I was hoping that this, coupled with the Central Limit Theorem, would mean that $N = 30$ measurements are sufficient to cause $\bar{x}$ to be approximately normal and a t-distribution CI to be reasonably accurate. $\endgroup$ – user200783 Jul 23 '18 at 14:17
  • $\begingroup$ However, from your answer, it seems that $N = 30$ may not be "large enough" - is there a recommended methodology (perhaps including a test for normality?) to determine how many measurements I should take before calculating a t-distribution CI? $\endgroup$ – user200783 Jul 23 '18 at 14:17

1. Normal data, variance known: If you have observations $X_1, X_2, \dots, X_n$ sampled at random from a normal population with unknown mean $\mu$ and known standard deviation $\sigma,$ then a 95% confidence interval (CI) for $\mu$ is $\bar X \pm 1.95 \sigma/\sqrt{n}.$ This is the only situation in which the z interval is exactly correct.

2. Nonnormal data, variance known: If the population distribution is not normal and the sample is 'large enough', then $\bar X$ is approximately normal and the same formula provides an approximate 95% CI. The rule that $n \ge 30$ is 'large enough' is unreliable here. If the population distribution is heavy-tailed, then $\bar X$ may not have a distribution that is close to normal (even if $n \ge 30).$ The 'Central Limit Theorem', often provides reasonable approximations for moderate values of $n,$ but it is a limit theorem, with guaranteed results only as $n \rightarrow \infty.$

3. Normal data, variance unknown. If you have observations $X_1, X_2, \dots, X_n$ sampled at random from a normal population with unknown mean $\mu$ and standard deviation $\sigma,$ with $\mu$ estimated by the sample mean $\bar X$ and $\sigma$ estimated by the sample standard deviation $S.$ Then a 95% confidence interval (CI) for $\mu$ is $\bar X \pm t^* S/\sqrt{n},$ where $S$ is the sample standard deviation and where $t^*$ cuts probability $0.025$ from the upper tail of Student's t distribution with $n - 1$ degrees of freedom. This is the only situation in which the t interval is exactly correct.

Examples: If $n=10$, then $t^* = 2.262$ and if $n = 30,$ then $t^* = 2.045.$ (Computations from R below; you could also use a printed 't table'.)

qt(.975, 9);  qt(.975, 29)
[1] 2.262157  # for n = 10
[1] 2.04523   # for n = 30

Notice that 2.045 and 1.96 (from Part 1 above) both round to 2.0. If $n \ge 30$ then $t^*$ rounds to 2.0. That is the basis for the 'rule of 30', often mindlessly parroted in other contexts where it is not relevant.

There is no similar coincidental rounding for CIs with confidence levels other than 95%. For example, in Part 1 above a 99% CI for $\mu$ is obtained as $\bar X \pm 2.58 \sigma/\sqrt{n}.$ However, $t^*=2.76$ for $n = 30$ and $t^* = 2.65$ for $n = 70.$

[1] 2.575829
qt(.995, 29)
[1] 2.756386
qt(.995, 69)
[1] 2.648977

4. Nonnormal data, variance unknown: Confidence intervals based on the t distribution (as in Part 3 above) are known to be 'robust' against moderate departures from normality. (If $n$ is very small, there should be no far outliers or evidence of severe skewness.) Then, to a degree that is difficult to predict, a t CI may provide a useful CI for $\mu.$ By contrast, if the type of distribution is known, it may be possible to find an exact form of CI.

For example, if $n = 30$ observations from a (distinctly nonnormal) exponential distribution with unknown mean $\mu$ have $\bar X = 17.24,\, S = 15.33,$ then the (approximate) 95% t CI is $(11.33, 23.15).$


        One Sample t-test

data:  x
t = 5.9654, df = 29, p-value = 1.752e-06
alternative hypothesis: true mean is not equal to 0
95 percent confidence interval:
 11.32947 23.15118
sample estimates:
mean of x 

However, $$\frac{\bar X}{\mu} \sim \mathsf{Gamma}(\text{shape}=n,\text{rate}=n),$$ so that $$P(L \le \bar X/\mu < U) = P(\bar X/U < \mu < \bar X/L)=0.95$$ and an exact 95% CI for $\mu$ is $(\bar X/U,\, \bar X/L) = (12.42, 25.16).$

qgamma(c(.025,.975), 30, 30)
[1] 0.6746958 1.3882946
mean(x)/qgamma(c(.975,.025), 30, 30)
[1] 12.41835 25.55274

Addendum on bootstrap CI: If data seem non-normal, but the actual population distribution is unknown, then a 95% nonparametric bootstrap CI may be the best choice. Suppose we have $n=20$ observations from an unknown distribution, with $\bar X$ = 13.54$ and values shown in the stripchart below.

enter image description here

The observations seem distinctly right-skewed and fail a Shapio-Wilk normality test with P-value 0.001. If we assume the data are exponential and use the method in Part 4, the 95% CI is $(9.13, 22.17),$ but we have no way to know whether the data are exponential.

Accordingly, we find a 95% nonparametric bootstrap in order to approximate $L^*$ and $U^*$ such that $P(L^* < D = \bar X/\mu < U^*) \approx 0.95.$ In the R code below the suffixes .re indicate random 're-sampled' quantities based on $B$ samples of size $n$ randomly chosen without replacement from among the $n = 20$ observations. The resulting 95% CI is $(9.17, 22.71).$ [There are many styles of bootstrap CIs. This one treats $\mu$ as if it is a scale parameter. Other choices are possible.]

B = 10^5; a.obs = 13.54
d.re = replicate(B, mean(sample(x, 20, rep=T))/a.obs)
UL.re = quantile(d.re, c(.975,.025))
    97.5%      2.5%
 9.172171 22.714980 
  • $\begingroup$ Thank you very much for this information. In particular, it looks like the answers to my initial questions are: $\endgroup$ – user200783 Jul 23 '18 at 14:17
  • $\begingroup$ 1. Since the population variance is unknown, the t-distribution value (e.g. $2.045$ for $N = 30$) is the correct value to use. However, for $N \ge 30$, the normal distribution value is a reasonable approximation. $\endgroup$ – user200783 Jul 23 '18 at 14:17
  • $\begingroup$ 2. If we can assume that the data are normally distributed, it is valid to calculate a CI using the t-distribution even if $N < 30$. If they are not normally distributed, a t-distribution CI may or may not be useful. It's interesting that the "Statistically Rigorous Java Performance Evaluation" paper specifically says that, even for small $N$, "computing [t-distribution] confidence intervals does not require that the underlying data is [...] normally distributed". $\endgroup$ – user200783 Jul 23 '18 at 14:18
  • $\begingroup$ I'll simplify further for normal data. If variance is known, use z; if variance is unknown, use t; ignore sample size. // For data of questionable normality, first ponder whether you know they're from some other distribution for which an exact test is known. If n small with no far outliers or obvious skewness or n large, maybe t is OK. Consider using rank-based test or bootstrap when in doubt. // 'Java' article mostly OK, but seems dated and derivative at points, would doubt it was refereed by a statistician. $\endgroup$ – BruceET Jul 23 '18 at 16:58

First, $\sigma \over \sqrt{n}$ is not standard deviation, it is standard error. Second, k depends on the confidence level you are interested in and, as you said, can be taken either from a normal distribution or t-distribution. Third, the t-distribution is meant mainly for small sample sizes ($n < 30$) and for large sample sizes it doesn't matter whether you apply the normal distribution or t-distribution as the two approximate. Finally, if $\sigma$ is not known, better to go with the t-distribution using the sample standard deviation instead of $\sigma$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.