How do I compare the repeatability of two sets of experiments with a low number of samples? I'm investgating the repeatability of an experimental process (actually measuring lift in a wind tunnel), which can be measured using a single number, Q. However, I only have limited access to the tunnel so the number of experiments that I can run is restricted.
If I have 5 experiments with the tunnel in one configuration:  Q1 = [1.18, -0.41, -0.66, 0.98, 0.1]
and five in another: Q2 = [-0.36 -0.73 -1.47 0.15 -0.31]
How confident can I be that the repeatability of the tunnel in the second configuration is better than the first? i.e. the standard deviation of the data from the first configuration is 0.81 and from the second configuration is 0.60, but these values will clearly change if I do additional experiments. Can I somehow calculate the standard deviation of the standard deviation?
Any advice on how I could calculate this would be appreciated and more generally how I can determine when I have enough experiments to say something like I am 95% confident that configuration 2 has better repeatability than configuration 1.
 A: Look up the two-sample F-test for variances. With equal sample sizes Q, this test calculates the ratio of the two variances, and then compares the value to the F-distribution with (Q-1, Q-1) degrees of freedom. A main assumption is normality, but that might be OK for random measurement error like this. 
With your values, however the difference is not statistically significant (p=0.29). In general, the standard error of the standard deviation is large unless you have lots of samples, and thus detecting differences in variability (repeatability) is difficult.
A: If you want a non-parametric approach that doesn't assume normality, you could obtain distributions of SDs for each configuration using bootstrapping, then compare these distributions. Comments below the answer to this question suggest that non-overlapping 84% confidence intervals would indicate a difference in the SDs at an alpha of .05. Alternatively, on each iteration of the bootstrap you could find the difference between the two pseudo-estimates of the SD then compare the distribution of this difference to zero. Below is R code achieving both approaches; both suggest that you have not observed a difference that should be considered real.
a = c(1.18, -0.41, -0.66, 0.98, 0.1)
b = c(-0.36, -0.73, -1.47, 0.15, -0.31)
library(plyr)
iterations = 1e4
sds = ldply(
    .data = 1:iterations
    , .fun = function(x){
        a_sd = sd(sample(a,replace=T))
        b_sd = sd(sample(b,replace=T))
        to_return = data.frame(
            a_sd = a_sd
            , b_sd = b_sd
            , diff = a_sd - b_sd
        )
        return(to_return)
    }
    , .progress = 'text'
)

quantile(sds$a_sd,c(.08,.92))
 quantile(sds$b_sd,c(.08,.92))
#substantial overlap

quantile(sds$diff,c(.025,.975))
#includes zero

A: I'm not sure if such a (non-parametric) permutation procedure could be applied here. Anyways, here is my idea:
a <- c(1.18, -0.41, -0.66, 0.98, 0.1)
b <- c(-0.36, -0.73, -1.47, 0.15, -0.31)
total <- c(a,b)
first <- combn(total,length(a))
second <- apply(first,2,function(z) total[is.na(pmatch(total,z))])
var.ratio <- apply(first,2,var) / apply(second,2,var)
# the first element of var.ratio is the one that I'm interested in
(p.value <- length(var.ratio[var.ratio >= var.ratio[1]]) / length(var.ratio))
[1] 0.3055556

