0
$\begingroup$

I'm interested in sampling a large dataset in nonconsecutive-record sequences of arbitrary length without overlap. I know from

How to take many samples of 10 from a large list, without replacement overall

how to do it if the sequence length is fixed (at 10 in that example), via 

sample <- split(sample(datapoints), rep(1:(length(datapoints)/10+1), each=10))

How do I generalize this to emit uniformly-distributed sized samplings, not just sequences of 10?

Furthermore, can I specify that I want N such sequences?

For example, suppose I have the sequence d <- 1:20 and permute it via sample(d, 20, replace = F) to obtain another permuted sequence. Now I want to extract arbitrarily sized subsequences of this permuted sequence, say d1 <- c(1,5,4,15,3), d2<- c(18,7,12,11,19,16,10,8,14,17, 20, 13), d3 <- c(9,2,6) in a quick manner as in the example. My dataset is large, and I'd like to simply sample it once, and then split it without any length constraint as in the post I cite.

|cite|improve this question
$\endgroup$
  • $\begingroup$ Are you familiar with the binomial distribution. You can randomly draw from a binomial to decide sample length and which chunks to draw. $\endgroup$ – Zachary Blumenfeld Dec 6 '15 at 0:34
  • $\begingroup$ Arbitrarily sized as in e.g. 25 instead of 10? $\endgroup$ – tho_mi Dec 6 '15 at 0:34
  • $\begingroup$ A uniform distribution of sizes. $\endgroup$ – user1809593 Dec 6 '15 at 0:35
  • $\begingroup$ Not sure I understand. The sampling has to be exhaustive, i.e. include each datapoint exactly once. $\endgroup$ – user1809593 Dec 6 '15 at 0:37
  • $\begingroup$ You need some process to decide your sample length and which data to sample. If you would like these things to not be fixed but rather allow them to vary randomly you must draw them from a descrete probability distribution. The binomial distribution is a good place to start $\endgroup$ – Zachary Blumenfeld Dec 6 '15 at 0:45
1
$\begingroup$

Here's a simple answer based on Zachary's comment. However, for larger datasets (dim(data1)[1]) it isn't efficient and plus it doesn't simply permute the dataset once via sample and then split it into arbitrarily sized samples, which was the elegant logic of the original post I cited above. 

#data1 <- 1:14e6
#data <- sample(data1, length(data1), replace = F) #not necessary
data <- 1:14e6
K <- 0
rand.samp <- NULL
while(dim(as.matrix(data))[1] != 0) {
K <- 1 + K
n=length(data);
nn=sample(1:n,n,replace=FALSE);
rand.start=sample(1:(n-1),1);
rand.end=sample((rand.start+1):n,1);
rand.samp[[K]]=data[nn[rand.start:rand.end]];
data <- setdiff( data, unlist(rand.samp))
print(dim(as.matrix(data))[1])
                                    }

desired output is this list

rand.samp

histogram of random sequence lengths

hist(log10(unlist(lapply(rand.samp, length))), breaks = 100)
|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.