1
$\begingroup$

If I have the mean, s.d., median and count for sample A, and the same for sample B, can I throw away the samples, and calculate exactly mean, median and s.d. for the combination? Well the mean is easy.

Here is the question in R code:

a=c(1,2,3,4,5)
b=c(2,2,3,4,4,5)
c=c(a,b)
m=( (length(a)*mean(a)) + (length(b)*mean(b)) ) / (length(a)+length(b))
print(mean(c) == m) #TRUE
s= ???
print(sd(c) == s)
d= ???
print(median(c) == d)

I tried a few things for s.d. but failed; I had no idea for the median. (For s.d. the closest I got was var(a)*(length(a)-1), doing the same for b, then dividing the sum by length(a)+length(b)-1. That gives 1.733 compared to var(c) of 1.7636.)

As a practical example, if I'm collecting stock ticks, I calculate mean, median and sd for each 1 minute period. Can I then use those 1 minute bars to make the same data for the 5 minute bars, and use the 5 minute bar data to make the hourly bars, and so on? Or, if I want to know the s.d. for weekly bars do I need to keep the ticks and load a full weeks worth of ticks in memory? (Yes, I realize I could get a good enough approximation of the weekly answer by treating my minute bars as ticks, but I wanted to confirm my hunch that an exact answer is impossible once I've thrown away the ticks.)

UPDATE: given that I was wrong and getting the s.d. is possible (see answers below), I'm now wondering if the median is also not impossible? E.g. how about if I also knew the median absolute deviation (mad() in R) for each sub-period?

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ You can figure out the value $\sum_i(a_i-\bar{a})^2$ and $\sum_j(b_j-\bar{b})^2$ from the given s.d.'s and known sizes of the two data sets, and therefore can figure out $\sum_ia_i^2$ and $\sum_jb_j^2$. What you are after is $\sum_i (a_i-\bar{a+b})^2+\sum_j(b_j-\bar{a+b})^2$ where you know $\bar{a+b}$ and can creatively use $(\alpha-\beta)^2=\alpha^2+\beta^2+2\alpha\beta$ to get this quantity. Divide by size of total data set $- 1$, take square root, and you have the s.d. of the combined data set. Sorry, can't help you with the R code; I am not a statistician. $\endgroup$
    – Dilip Sarwate
    Commented Dec 5, 2011 at 1:35
  • $\begingroup$ Those overbars were supposed to extend over the entire quantity $a+b$ but seem to have gotten shortened for some reason and it is too late to edit my comment: five minutes have passed, $\endgroup$
    – Dilip Sarwate
    Commented Dec 5, 2011 at 1:40

4 Answers 4

Reset to default
3
$\begingroup$

I'm too lazy to write out the math but I already have some R code that reconstructs the variance of the combined datasets given the variances and means from the subsets.

dat1 <- rnorm(20)
dat2 <- rnorm(10)
dat <- c(dat1, dat2)

n <- length(dat)
n1 <- length(dat1)
n2 <- length(dat2)

m1 <- mean(dat1)
m2 <- mean(dat2)
m <- mean(dat)  # We know we could reconstruct this

# Only a function of the means and variances and lengths of the subsets
((n2-1)/(n-1))*var(dat2) + n2*(m2-m)^2/(n-1) + ((n1-1)/(n-1))*var(dat1) + n1*(m1-m)^2/(n-1)
# But it matches the variance of the combined data
var(dat)

I'll note that the reason your attempt doesn't work is that you fail to account for idea that the combined variance is calculated with the combined mean whereas the variances for the subsets calculated with their respective means. The piece that I add in are the differences between the subset means and the combined mean scaled appropriately.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Wow, thanks! Confirmed it works. I made cf=function(x,m,n){nx=length(x);( ((nx-1)*var(x)) + (nx*(mean(x)-m)^2) ) / (n-1)} which is the contribution from each period. Used like this: m=mean(c);n=length(c);s=sqrt( cf(a,m,n) + cf(b,m,n) ) $\endgroup$
    – Darren Cook
    Commented Dec 5, 2011 at 2:27
  • $\begingroup$ No problem. Wish I could help more but I haven't put much thought into the median problem. My guess is that the median isn't completely recoverable in general but it might be in some very special cases. $\endgroup$
    – Dason
    Commented Dec 5, 2011 at 2:33
  • $\begingroup$ "Used like this: m=mean(c);n=length(c);s=sqrt( cf(a,m,n) + cf(b,m,n)" I thought the idea was to throw away all the data points in a and b and retain only $\mu_a$, $\mu_b$, $\sigma_a$, $\sigma_b$, and $n_a$, $n_b$, the number of points in each set. So, m=mean(c) seems to not conform to this ideal. Why not set $n=n_a+n_b$, $$\mu=\frac{n_a*\mu_a+n_b*\mu_b}{n}$$ etc and thus get all the information from the summary statistics without requiring all the data points? $\endgroup$
    – Dilip Sarwate
    Commented Dec 5, 2011 at 17:24
  • $\begingroup$ @DilipSarwate Yes, you're right. I was being brief, showing what m and n are; they'd be calculated the way you describe when c is not around any more. $\endgroup$
    – Darren Cook
    Commented Dec 9, 2011 at 9:59
3
$\begingroup$

Take a look at the formulas for calculating the various sums of squares and mean squares for a one-way ANOVA. What you want to do is just running ANOVA backwards.

Improve this answer
$\endgroup$
1
  • $\begingroup$ This is very true. It wasn't until after I originally wrote the code that I submitted (which I wrote quite a while ago) that I realized the connection and felt a little silly for not realizing it sooner. $\endgroup$
    – Dason
    Commented Dec 5, 2011 at 2:23
3
$\begingroup$

Darren,

There are some 'running' algorithms for fast medians which use binning. I looked at a few in the past but have nothing ready-made I can point you to. This thread at stackoverflow maybe be useful.

Improve this answer
$\endgroup$
1
1
$\begingroup$

To calculate the combined/composite median, you need all the individual measurements. Sorry, that's the only way to do it.

But it is possible to correctly calculate the combined/composite standard deviation without having all the individual measurements. You need three numbers for each sub-period: the mean of the measurements taken in that sub-period, the standard deviation of the measurements taken in that sub-period, and the number of measurements which were taken during that sub-period. This web page describes how you can then correctly calculate the standard deviation of the whole dataset; it includes source code in Perl do do it: http://www.burtonsys.com/climate/composite_standard_deviations.html

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.