What information is required to aggregate percentiles I have several thousand large datasets that are too big to fit into memory at once, so I need to keep them separate.
It is easy enough to get the count, mean, std dev, min and max for the whole dataset, but I also need the percentiles for P1, P5, P10, ..., P90, P95, and P99.
I can generate these numbers (or basically any stat) for each individual dataset. If I have this information for each individual dataset can I then combine the information needed post facto?
I have read many similar questions on here, but I feel I have an advantage since I do have all the original data and not just a small number of summary stats.
Note all datasets have different sample sizes, and distribution is not necessarily normal.
 A: It turns out there is a vast body of literature focused on this problem in the context of streaming data.
This site provides an excellent overview of several methods.  In addition there are libraries such as DDSketch and T-Digest that handle this exact category of problem.  None of the solutions are exact, but they do guarantee accuracy within a given tolerance, and you can reduce that by keeping more samples (bit still  trivial number relative to the total number of samples).
A: As I understand it, you want to combine all the datasets together and get the percentiles for the entire dataset (single variable?), but it doesn't fit in memory.
I can't think of any way that one can "aggregate" percentiles so I would approach it as a technical problem.
Depending on the tech / software you're using, are there any tools to sort the variable in-place? Is it possible to increase the virtual memory?
If not, you may have to think of ways to manually hack your data into batches and preprocess it step by step, without loading everything into memory (I used to do this a lot for particle physics experiments with insanely huge datasets).
For example, do you have a lot of repeated values? Then you could think of a kind of compression - scan the datasets one by one and store the counts for each value you encounter (rather than the values themselves).
If that's not the case, you could estimate some "breakpoints" in your data, and start scanning the datasets and splitting them up and writing them into roughly "ordered" datasets. Once you have the rough datasets, you can do some more sorting & refining on that until you get something that's ordered and fits in memory nicely so you can get P1 to P5... then move onto the next part of the data, and so on.
Or, depending on how accurate your results need to be, you could make the data more coarse by reducing e.g. the decimal precision (making it more easy to count similar values), or some other sort of "binning" (like a histogram, though it would be a fine-tuning job to choose such a binning that can give you reasonably accurate percentiles).
It's a bit vague but that kind of thing (essentially, pre-processing steps to cut down your data volume while retaining the essential info) might help.
A: You can't aggregate percentiles without loss of accuracy. As long as you know the number of samples that went into each set of percentiles, you can approximate the combined percentiles reasonably well, however.
This could be done several ways. Since information has been lost when computing percentiles, you have to make some kind of assumption to approximate the original data from the percentiles and then calculate new combined percentiles. One alternative would be to assume that data points are only present at the percentiles (and accept fractional counts). Another would be to assume that the data is uniformly spread out between percentiles in some way.
Let's go with the first assumption as the base for our approximation. If you have two sets of percentile measure available with a known count of samples for each, then you have the values of the percentiles for each and you can assign a frequency of 1/100th of the total count (which may not be an integer). Now, you just sum the total counts and calculate the number of samples per percentile in the new aggregate. Starting from the bottom, you then go through the combined data points (lower to higher) and assign them to your new percentiles until each fill up and then continue to the next and so on. The new percentiles will line up perfectly with a subset of the original percentiles with this method.
Another option would be to instead summarize each data set as a much more fine-grained histogram. The same method can be used based on that and the buckets can be made to line up between all the data sets from the start. This will work the same, but errors can be made smaller.
The approximation errors will depend only on actual samples being approximated incorrectly by at most one bucket's width. In other words, the smaller buckets (size of percentiles or similar) you start out with, the smaller the error. The error can be made arbitrarily small by using more memory.
