Business Problem

We are measuring the duration of an operation on our server. We want to know if certain changes (call them treatments) make this operation faster or slower.


Our current test harness is outputting data every 3 seconds. During those 3s it is collecting a large number of individual trials (30-70) and outputting the mean, max, min and variance.


Can we treat each 3s period as a trial and use the mean from each period as a trial for Student's t-test, Welch's and the other statistical methods for doing A/B testing?


My feeling is that this is not statistically valid and that we need to use each individual trial. This does generate a lot of raw data so we'd like to confirm this before we do this.

Many thanks!


Using pre-aggregated data changes the appropriate interpretation of the results. It is not problematic from the perspective of calculation or the assumptions of the models you propose. However, your estimates will increase in accuracy as your number of observations rises. Pre-aggregating thus increases the range of your confidence intervals.

You are also implicitly building in an additional assumption with this approach, namely that all of the observations within a 3s interval are distributed similarly across all such intervals (and are well described by a mean). More simply, that each of your means is a good summary of that interval. If you looked at all of the trials, you could analyze the distribution of your measurements without this assumption.

This discussion might also be interesting.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.