Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I have data from a load test of an automated system with several thousand data points that cover roughly 1 week of operation. I need to compare several algorithms for this system to see which is faster, if any (the metric used is the response time of the algorithm in milliseconds). Since it's technically impossible to store the response time of each operation of the algorithm, the simulator computes the mean, stdev and sample count of all the operations performed during the last minute, in order to reduce the amount of data generated.

Question 1:

Is it possible (or wise) to estimate the global mean/stdev based on the mean/stdev of each sample? Since each sample has a different sample count, I assume that some kind of weighting should be used for this estimation, but I'm not sure how to do this robustly, and I don't want to inadvertently alter the analysis.

Question 2:

A coworker suggested that computing the global quartiles (Q1, Q2, Q3) from the samples might be a more sensible approach to convey the general behavior of each algorithm. Which method should be preferred?

I'd like to ensure that the analysis is statistically robust and that I'm not seeing "things that aren't there".

UPDATE 1:

I've found out that the global mean can be estimated by computing a weighted mean of the means of the samples and the global stdev can be obtained using the pooled variance estimate. Is this correct?

Nevertheless, I still don't know if it is possible to do the same with the median as I haven't found much literature regarding weighted medians. I still don't know which measure is more appropriate for my data.

UPDATE 2:

I've learnt that Quartiles should be preferred to mean and standard deviation when data does not follow a normal distribution, although I don't know how to determine if my data follows it or not. I've found out some methods to compute the weighted median that I assume can be applied to any quartile but ¿does it make any sense to compute the weighted median from the collected means?

share|improve this question
    
I'm a bit confused about why your questions are about the global mean and standard deviation but your main research question is about the difference between different samples. Can you clarify that this is correct, and if so, say why you need the global mean and standard deviation? –  Peter Ellis Jan 22 '13 at 18:36
add comment

1 Answer 1

up vote 1 down vote accepted
+50

Edit - this answer misses the point of the question - see the comments below. I'm only leaving it here in case anyone else misinterprets it like me.

Your question 2 first.

To assess whether your population is normally distributed or not, use graphical methods. Standard methods include a histogram, a line plot showing an estimate of the density, and a qq plot comparing to a theoretical distribution. The first two give you an intuitive feel for the shape of the data and you can use a different colour line to show the normal distribution as a reference point. The qq plot takes a bit of practice to interpret but is the best method for comparing to an actual distribution. I'd use all three (or at least the density line plot and the qq plot) in combination. Any stats package will let you do this.

Edit/addition: In your case as the observations are taken sequentially you should also draw a straightforward time series plot to give a sense of whether it is stationary or moving over time, and there is a range of basic time series diagnostic plots that you should look at too. If there is a structural trend in your observations over time the whole approach will need to be thought through very carefully.

You are correct that when populations are not normally distributed often the mean is not a good summary measure. The main reason is that it can be heavily influenced by a single outlier. A good summary measure for your purposes is probably a 20 percent trimmed mean - which cuts the top and bottom 20% off your data and takes the mean of what is left. A median is the same as a 50% trimmed mean and, while less influenced by outliers than the mean, often goes too far in discarding information in the bulk of the data.

In comparing the trimmed means of your various sub samples you should use a bootstrap method to quantify how much uncertainty there is in the various estimates from the different sub samples.

The graphical methods I mentioned above in assessing your distribution against normal can also be used to compare the distributions of your various sub samples. A good addition is a boxplot, which takes an approach similar to that suggested by your colleague. A range of graphics to explore the data, in combination with comparing the trimmed mean and using a bootstrap for inference, is probably what you need.

Your question 1. It's not clear to me why you need the global mean and standard deviation and what they would be used for (see my comment). I may come back and edit this question if I understand this better. As it is, you need to think through what you mean by a global mean, given you are (if I understand the situation correctly) manipulating the system yourself.

share|improve this answer
    
Thanks for your reply. I need to assess the performance of each algorithm based on their results for 1 week. Every minute of simulation I get the mean+stdev of the response time spent for all requests during that minute. My concern is that I need a way to infer/compute the overall performance from the collection of 1 minute means+stdevs. Since 1 minute can have 10 requests and the next 100, they don't have the same relevance in the overall behavior. My reasoning is that I should account for that with some kind of weighting, hence my idea of using a weighted mean/median. –  Alberto Miranda Jan 22 '13 at 19:13
    
ah, ok, i completely missed the point. I thought you had a sub sample of the actual responses. –  Peter Ellis Jan 22 '13 at 20:00
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.