Computing the difference between means and resolving a new/composite sample size When computing the difference between means I can resolve a new mean, variance, standard error of the mean, and margin of error, but is there a way to compute a new/composite sample size value?
I ask because I use the difference between means as a way to calibrate the mean and later when comparing other calibrated means sample size comes into the formulas.
(Source)
Update
In the code example below I subtract the calibrations mean from the benchmarks mean time to execute a test. The calibration is an empty test to clock the cost of the benchmarks overhead.
The only part I am missing is how to create a composite sample size (if that is possible).
       mean -= cs.mean;
       variance += cs.variance;
       sd = sqrt(variance);
       moe = sd * getCriticalValue(getDegreesOfFreedom(me, cal));
       rme = (moe / mean) * 100;
       extend(me.stats, {
         'ME': moe,
         'RME': rme,
         'deviation': sd,
         'mean': mean,
         'size': ???,
         'variance': variance
       });

 A: I just think you are making it just too complex -- it is a JS benchmark for other programmers, not a clinical trial or Higgs boson search that will be peer reviewed by bloodthirsty referees and later have a great impact.
Just make a non-small number of repetitions (30), of both test and empty test, subtract the means, calculate error of this mean difference as I wrote you before and publish just this -- adding dozens of other measures will only confuse the readers.
Then, for each comparisons, you will have two means $X$ and $Y$ and two errors of that means, respectively $\Delta X$ and $\Delta Y$, so calculate $T$ as
$$T=\frac{X-Y}{\sqrt{(\Delta X)^2+(\Delta Y)^2}},$$
assume $\infty$ degrees of freedom for t-test and you're done. (If you don't believe that $30=\infty$, see this.)
A: How are you comparing the calibrated means? If you're looking at differences between them, and testing if that's zero using a $t$-test, then surely the mean of the empty loop ($\bar{x}_0$, say) will simply cancel out:
$$(\bar{x}_1 - \bar{x}_0) - (\bar{x}_2 - \bar{x}_0) = \bar{x}_1 - \bar{x}_2$$
A: Thanks @mbq and @onestop. After running some tests the calibration was a wash. Subtracting the means raised the margin or error to the point that the calibrated and none calibrated test results where indeterminately different.
@mbq I will take your advice and reduce the critical value lookup to 30 (as infinity).
When comparing against other benchmarks I think I can even avoid the walsh t-test because the variances are so low (all below 1). Some examples are the variances of benchmarks are:
1.1733873442408789e-16, 0.000012223589489868368, 3.772214786601029e-19, and 6.607725046958527e-16.
