Validity of using the mean duration across a sample period as the trial for Student's t test or Welch's test

Question

Business Problem

We are measuring the duration of an operation on our server. We want to know if certain changes (call them treatments) make this operation faster or slower.

Data

Our current test harness is outputting data every 3 seconds. During those 3s it is collecting a large number of individual trials (30-70) and outputting the mean, max, min and variance.

Question

Can we treat each 3s period as a trial and use the mean from each period as a trial for Student's t-test, Welch's and the other statistical methods for doing A/B testing?

Discussion

My feeling is that this is not statistically valid and that we need to use each individual trial. This does generate a lot of raw data so we'd like to confirm this before we do this.

Many thanks!

Answer 1

Using pre-aggregated data changes the appropriate interpretation of the results. It is not problematic from the perspective of calculation or the assumptions of the models you propose. However, your estimates will increase in accuracy as your number of observations rises. Pre-aggregating thus increases the range of your confidence intervals.

You are also implicitly building in an additional assumption with this approach, namely that all of the observations within a 3s interval are distributed similarly across all such intervals (and are well described by a mean). More simply, that each of your means is a good summary of that interval. If you looked at all of the trials, you could analyze the distribution of your measurements without this assumption.

This discussion might also be interesting.