I have the following problem:
(1) I get a package with many samples with values (2) I have reasonable time to calculate whatever i want and keep in memory any info i want, but not all of the samples (the samples are very large, I can keep a few MB of data) (3) I drop the package and go to step (1) with a new package, unless no more packages (4) Use all kept information to calculate the 10% percentile \ median \ 90% percentile
Can it be done? Thx
You don't describe the process by which you are receiving the data well enough for a solid answer. What follows is speculation that might be helpful.
If it is feasible to take a random sample of 100 values from among a huge number of values (say 100,000) you might get sufficiently close.
Suppose you have 100,000 values that happen to be distributed $\mathsf{Norm}(100, 15).$ Then the 10th, 50th and 90th percentiles of the population are about 81, 100, 119, from R as follows:
qnorm(c(.1,.5,.9), 100, 15)
[1] 80.77673 100.00000 119.22327
If you were able to take 100 observations at random from among 100,000, you might get approximate respective percentiles 83, 100, 119, as below:
set.seed(2020)
x = rnorm(10^5, 100, 15)
s = sample(x, 100)
quantile(s, c(.1,.5,.9))
10% 50% 90%
82.71281 99.60697 118.63415
Not perfect but perhaps good enough for practical purposes.
In you have any idea that batches may be random samples from the entire population of values, it might suffice to find these quantiles from the first 'package'. Then do the same for a few additional 'packages' to see if results are similar from one 'package' to the next.
For example, if the values are exponentially distributed with mean 100, and you you look at 1000 values from each of several packages you might get results similar to those below. I chose to look at exponential data for this example, because sample quantiles (especially the 90th) are more variable than for normal data.
set.seed(1122)
quantile(rexp(1000, 1/100), c(.1,.5,.9))
10% 50% 90%
12.46320 67.56755 235.70446
quantile(rexp(1000, 1/100), c(.1,.5,.9))
10% 50% 90%
11.00286 78.51803 236.77132
quantile(rexp(1000, 1/100), c(.1,.5,.9))
10% 50% 90%
11.05049 74.76116 240.70067
quantile(rexp(1000, 1/100), c(.1,.5,.9))
10% 50% 90%
9.464616 67.543288 248.137799
The respective population quantiles are about 11, 69, and 230.
