0
$\begingroup$

So I estimated a particular statistic $\Phi$ (custom made) by bootstrapping 1000 samples from the original dataset to generate 1000 different $\Phi$s. The issue is that saving those 1000 bootstrap statistics was not at all memory efficient, so I decided to just save the summary statistics from the bootstrap samples. Saving the samples is also difficult because there are lot of iterations of the bootstrap.

So for example, for each iteration, I have 1000 samples of my dataset, I get 1000 different $\phi$, from which I calculate the summary (mean, median etc).

I would eventually report the summary statistic, but I also have to analyze the distribution of my bootstraps.

I have saved the mean, standard deviation, median, first and third quartiles, the min and max as well as the 5th and 95th percentiles. I do presume that the bootstrap distribution should look normal. But I wanted a robust way to generate my samples back (1000 samples) for further analysis based on these summary statistics.

Here is what I have tried so far. Assuming the bootstrap sampling distribution to be normal, I tried using the truncnorm distribution in Python to specify the min,max,mean and std deviation and generate 1000 samples. Then I find the index of the corresponding percentiles (medians and the others) in those 1000 samples, and just change them to the summary statistics I have.

I tried searching different StackOverflow forums for an answer but this is the best I could come up with till now.

It would be helpful if I can get further insights on this.

Improve this question
$\endgroup$
9
  • 1
    $\begingroup$ If you have access to the original data and know the random seed that was used, generating the samples is a deterministic process that can be repeated. $\endgroup$
    – Tim ♦
    May 31 at 10:12
  • $\begingroup$ @Tim yes, computer generated "random" numbers are indeed deterministic! That is why i suggested setting the seed in my answer :-) just didn't want to reveal the brutal truth of deterministic random numbers so directly... $\endgroup$
    – Ute
    May 31 at 10:17
  • $\begingroup$ Why do you "find the index of the corresponding percentiles (medians and the others) in those 1000 samples, and just change them to the summary statistics I have."? If you save the summary statistics in a separate array, you would not need much more memory. $\endgroup$
    – Ute
    May 31 at 10:24
  • $\begingroup$ This is for re-creating the samples of my custom statistic. The samples are not indices but rather a complex matrix operation that I do on on my dataset. The samples have resampled indices on which I calculate my custom statistic. I did indeed set a seed, but why will repeat the calculation of my custom statistic all over again? It takes over 2 days for my big dataset. $\endgroup$
    – cwanderroycbooks
    May 31 at 14:04
  • $\begingroup$ @cwanderroycbooks, your last comment is actually pretty important to understand you problem. Maybe you could add this to your question. $\endgroup$
    – Ute
    May 31 at 14:22

1 Answer 1

Reset to default
2
$\begingroup$

It is not feasible in general to try to retrieve the bootstrap samples from summary statistics: there is simply not enough information in them. Therefore you need some other way to keep information about the actual bootstrap samples, or to regenerate them quickly.

1. approach: keep bootstrap sample information

In a situation where the problem is that data items are "bulky", I'd split the bootstrapping into two steps. This requires that you have a dataset consisting of identifiable items (rows), and that you can pick the $i$-th element easily.

1.) generate samples of indices that point to elements from the original sample to be included in a particular bootstrap sample. You can also save these on disc if you need to repeat the experiment multiple times, and free / reuse memory.

2.) select the bootstrap samples one by one from the original data according to indices that were generated in step 1.), and calculate the complicated summary statistics.

2. approach: regenerate bootstrap sample information

Alternatively, if saving indices on disk is not an option, regenerate the samples from scratch, starting with the same state of random number generator (rng). In python, use random.seed, in R:set.seed. Then you can strip your code from all time consuming calculations of summary statistics and should retrieve the same bootstrap samples again in very short time.
Caveat!!: this approach only makes sense if calculating the summary statistics does not involve further simulations, such as MCMC. Simulation based statistics call the random number generator again. This would advance the rng state uncontrollably between bootstrap resampling steps.

Improve this answer
$\endgroup$
8
  • $\begingroup$ That's the issue right. Estimating the statistic itself takes 2 days in the server. I can try saving the samples but the memory will fill up quickly. That's why I am saving the bootstrap statistics. I don't see how saving the indices of the bootstrap helps. I would anyway have to re-calculate my custom statistic all over again to get my samples. What's the point of wasting 2 days? $\endgroup$
    – cwanderroycbooks
    May 31 at 14:01
  • $\begingroup$ No, you could save the indices, and the statistic separately. To reconstruct the samples, you do not need to estimate the statistics. Say, if your original sample has 5 items, then you can save the indices of the bootstrap sample, which could be (1,5,3,1,2), (4,4,1,3,2), etc. If you save these indices, you can also use them to afterwards calculate the statistics. I understand your question as: you are replacing some values in the bootstrap sample with the statistics you calculated. $\endgroup$
    – Ute
    May 31 at 14:10
  • $\begingroup$ Disclosure: I do use approach 1) myself sometimes, when dealing with time consuming statistics. And I (almost) always save the start value for the random number generator together with bootstrapped or simulated, just in case. For an R user, time can really be an issue. $\endgroup$
    – Ute
    May 31 at 14:57
  • $\begingroup$ @cwanderroycbooks If you already have spent a lot of computing time on the statistics, then use approach 2 - you know the start value of your random number generator. I hope in that case that the complicated statistics did not envolve extra random number generation in a black box (MCMC package or so). Then approach 2 would be wrong, because every calculation of the statistics advances the random number generator in an uncontrolled way. $\endgroup$
    – Ute
    May 31 at 15:01
  • $\begingroup$ Sorry I think there is a misunderstanding. I am not worried about saving the samples, but rather my statistic. Suppose I am calculating a statistic A (using complex linear algebra) from a dataset. Now to bootstrap, I generate 1000 different realizations of that dataset, which generates 1000 different As. Since I cannot save the As, I rather save their summary. $\endgroup$
    – cwanderroycbooks
    Jun 1 at 2:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.