I have a range of data (server performance statistics) is formatted as follows, for each server:

Time            | Average |  Min  |  Max  | StdDev  | SampleCount |
-------------------------------------------------------------------
Monday 1st      |    125  |   15  |  220  | 12.56   |     5       |
Tuesday 2nd     |    118  |   11  |  221  | 13.21   |     4       |
Wednesday 3rd   |    118  |   11  |  221  | 13.21   |     3       |
....            |    ...  |   ..  |  ...  | .....   |     .       |
and so on...

These data points are calculated from data that has a finer resolution (e.g. hourly data).

I need to aggregate this data into a single summary point so the end result is a list of servers and an aggregate average, min, max, standard deviation.

For average, I take the average of all the averages. For min, we take the minimum min. For max, we take the maximum max.

However, I'm not sure what method I should be using to aggregate standard deviation? I've seen various answers including square roots and variance but I really need a concrete answer on this - can anyone help?

First, the best way to do any of this is to use the raw data

If you don't have the raw data then

Second, for the average you should not simply average all the averages. You need to weight by sample size. In R, e.g

group1 <- c(2, 10, 12)
group2 <- c(4, 10, 15, 50)
m1 <- mean(group1)
m2 <- mean(group2)
(ave.unweight <- (m1 + m2)/2)  #13.875  ... incorrect
(ave.weight <- (m1*3 + m2*4)/7) #14.71
(ave.true <- mean(c(group1, group2))) #14.71

Third, for the SD see this thread

  • 1
    Good points from @peterflom. I would only add that calculating the coefficient of variation (SD/Mean) would give you a scale invariant metric of variability that would be comparable across servers of varying types, capacities, etc. – DJohnson Nov 26 '15 at 14:32
  • Thanks for the info - I've gone ahead and implemented the weighted average and that's working awesome. :) Unfortunately still struggling with the SD... Is there any chance you're able to provide some sample R code for the SD as per your linked thread - I'm having difficulty reading and understand their formula. – Dave Clarke Nov 26 '15 at 14:42
  • There is probably a package that does it, but the formula doesn't look so hard. – Peter Flom Nov 26 '15 at 16:21
  • From their formula, they have m and n - I don't quite understand where the two data sets come from my data? – Dave Clarke Nov 26 '15 at 16:29
  • m is the sample size of one group, n is the sample size of the other and your data are $x_i$ – Peter Flom Nov 26 '15 at 16:46

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.