Estimate global mean and stdev from samples taken at regular intervals

Question

I have data from a load test of an automated system with several thousand data points that cover roughly 1 week of operation. I need to compare several algorithms for this system to see which is faster, if any (the metric used is the response time of the algorithm in milliseconds). Since it's technically impossible to store the response time of each operation of the algorithm, the simulator computes the mean, stdev and sample count of all the operations performed during the last minute, in order to reduce the amount of data generated.

Question 1:

Is it possible (or wise) to estimate the global mean/stdev based on the mean/stdev of each sample? Since each sample has a different sample count, I assume that some kind of weighting should be used for this estimation, but I'm not sure how to do this robustly, and I don't want to inadvertently alter the analysis.

Question 2:

A coworker suggested that computing the global quartiles (Q1, Q2, Q3) from the samples might be a more sensible approach to convey the general behavior of each algorithm. Which method should be preferred?

I'd like to ensure that the analysis is statistically robust and that I'm not seeing "things that aren't there".

UPDATE 1:

I've found out that the global mean can be estimated by computing a weighted mean of the means of the samples and the global stdev can be obtained using the pooled variance estimate. Is this correct?

Nevertheless, I still don't know if it is possible to do the same with the median as I haven't found much literature regarding weighted medians. I still don't know which measure is more appropriate for my data.

UPDATE 2:

I've learnt that Quartiles should be preferred to mean and standard deviation when data does not follow a normal distribution, although I don't know how to determine if my data follows it or not. I've found out some methods to compute the weighted median that I assume can be applied to any quartile but ¿does it make any sense to compute the weighted median from the collected means?

Answer 1

Edit - this answer misses the point of the question - see the comments below. I'm only leaving it here in case anyone else misinterprets it like me.

Your question 2 first.

To assess whether your population is normally distributed or not, use graphical methods. Standard methods include a histogram, a line plot showing an estimate of the density, and a qq plot comparing to a theoretical distribution. The first two give you an intuitive feel for the shape of the data and you can use a different colour line to show the normal distribution as a reference point. The qq plot takes a bit of practice to interpret but is the best method for comparing to an actual distribution. I'd use all three (or at least the density line plot and the qq plot) in combination. Any stats package will let you do this.

Edit/addition: In your case as the observations are taken sequentially you should also draw a straightforward time series plot to give a sense of whether it is stationary or moving over time, and there is a range of basic time series diagnostic plots that you should look at too. If there is a structural trend in your observations over time the whole approach will need to be thought through very carefully.

You are correct that when populations are not normally distributed often the mean is not a good summary measure. The main reason is that it can be heavily influenced by a single outlier. A good summary measure for your purposes is probably a 20 percent trimmed mean - which cuts the top and bottom 20% off your data and takes the mean of what is left. A median is the same as a 50% trimmed mean and, while less influenced by outliers than the mean, often goes too far in discarding information in the bulk of the data.

In comparing the trimmed means of your various sub samples you should use a bootstrap method to quantify how much uncertainty there is in the various estimates from the different sub samples.

The graphical methods I mentioned above in assessing your distribution against normal can also be used to compare the distributions of your various sub samples. A good addition is a boxplot, which takes an approach similar to that suggested by your colleague. A range of graphics to explore the data, in combination with comparing the trimmed mean and using a bootstrap for inference, is probably what you need.

Your question 1. It's not clear to me why you need the global mean and standard deviation and what they would be used for (see my comment). I may come back and edit this question if I understand this better. As it is, you need to think through what you mean by a global mean, given you are (if I understand the situation correctly) manipulating the system yourself.