1
$\begingroup$

I am trying to aggregate data for testing for multiple data sets. It is unrealistic to load the entire data set (which is composed of multiple subsets of varying length) into memory to calculate a median. As one solution to my problem, I thought I could find the median of each subset of data, along with the sample size, and aggregate it into a weighted average by doing something like

(x_1(n_1) + x_2(n_2))/n

were x is the median and n is the data set size.

This is essentially taking a weighted average of medians. Is there a risk that taking the weighted average of medians will be far off from the actual median of the data set?

I am not a mathematician by trade so I appreciate the help. ty!

$\endgroup$

1 Answer 1

0
$\begingroup$

It depends on what you mean by "far off" and the nature of your data

  • If subsets of the data are nearly symmetrical and do not have long tails in either direction, then their means and medians will not be much different, and your method will give something like an ordinary weighted average (of means).

  • However, if some subsets are markedly skewed or long-tailed, as in the example below, then your weighted median might be relatively "far" from the median of the whole sample.

Fictitious data:

set.seed(2022)
x1 = rexp(20, 1/5)    # right-skewed
x2 = rt(10, 4)        # long tailed
x3 = rnorm(30, 20, 3)
x = c(x1, x2, x3)     # combined sample
median(x)
[1] 15.35097          # actual sample median
(20*median(x1) + 10*median(x2) + 30*median(x3))/(20+10+30) 
[1] 11.1669           # weighted sample median

boxplot(x1,x2,x3, varwidth=T)
 abline(h = 11.17, col="red")
 abline(h = 15.35, col="green2")

In the figure below the horizontal green line is the median of the sample, and the red line is your weighted median.

enter image description here

$\endgroup$
1
  • 1
    $\begingroup$ This is financial data so it is definitely not normally distributed, lots of long tail events. Thank you for your answer and demonstration - very helpful. $\endgroup$ Mar 12, 2022 at 13:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.