We are measuring the duration of an operation on our server. We want to know if certain changes (call them treatments) make this operation faster or slower.
Our current test harness is outputting data every 3 seconds. During those 3s it is collecting a large number of individual trials (30-70) and outputting the mean, max, min and variance.
Can we treat each 3s period as a trial and use the mean from each period as a trial for Student's t-test, Welch's and the other statistical methods for doing A/B testing?
My feeling is that this is not statistically valid and that we need to use each individual trial. This does generate a lot of raw data so we'd like to confirm this before we do this.