I have data generated by a RCT with a control and treatment group, each with n=300, so N=600. The observations are assumed to be i.i.d..
From that population, I'd like to draw 5 random observations without replacement (5 by group, so 10 in total). If I repeat and simulate this process say 1,000 times, are these 1,000 samples correlated with each other? Does this turn inference invalid, or say, biased?
My intuition tells me that there could me correlation among the 1,000 samples. The severeness of the issue most likely depends on the size of the randomly drawn sample. E.g., if I drew 200 random observations without replacement for each group instead of 5, and repeated this procedure 1,000 times, the observations ending up in the 1,000 simulated samples will be, clearly, very often the same one's. The 1,000 drawn samples will therefore correlate.
Thus, is there a rule of thumb about how large the number of repetitions should be in relation to the randomly drawn subsample out of the original population?
To respond to EdM: I simplified the problem a bit, but I think (and hope) it doesn't change much to my underlying concern about correlated "repetition samples". Here's why: The relevant question is: Within each treatment group, how many observations would I need to draw until I find 2 observations with the very same characteristic? (In real, the characteristics of an observation become only "apparent" once the observation is "drawn" so ex-ante the characteristics of the not yet drawn population is unknown. The drawing process - draw until 2 observations match - I want to mimic here is exactly the one happening in nature.) The average number of needed draws is roughly 5 observations. So I have on average 5 observations per treatment group entering each randomly drawn sample in each repetition. Yet, some characteristics are rarer, and there I end up with drawing on average 90 observations by treatment group to find 2 observations that match. Neglecting for now treatment differences.
To respond to EdM's answer: I am not sure this distribution applies in my setting. To me, it seems that in the Negative hypergeometric distribution it is in advance defined what is a "fail" or "success". So in advance, we define what a "match" or "failure" should be for a particular characteristic, e.g. Male/Female.
The process I want to mimic works differently in nature. It starts to draw an observation. It continues to draw observations until 2 observations exhibit the same characteristic, but this characteristic is not pre-defined in advance. The only thing that matters is that in the pool of drawn observations, 2 observations exhibit the very same characteristic. The exact peculiarity of the characteristic does not matter. Example: Let's assume we want to match on variable 1 (a certain dimension of possible characteristics):
- Draw 1 : AABC
- Draw 2: AACB
- Draw 3: CBAA
- Draw 4: CBAB
- Draw 5: AACB
- Stop. Draw 5 and draw 2 is a "failure".
Number of draws: 5. Thus, 2 observations within the pool of already drawn observations need to "match" or "fail", but the underlying value on what they are matched does not matter. E.g., Draw 5 could have been CBAB, too.
(Regarding the bootstrap, in this case, bootstrapping isn't very appropriate imho since in nature, the process does not work "with replacement".)
Yes, it is kind of a verification mechanism. Let me try to illustrate with a hopefully easy-to-understand “human” analogue: Assume Google has some new CAPTCHA’s to verify, where google does not know the solution yet. Google assumes users response to be true once 2 users agree. Users solve these CAPTCHA’s sequentially. User 1 arrives first, solves CAPTCHA number 1 and provides solution ABCDEF. User 2 arrives, provides for the same CAPTCHA another solution, say FABCDE. No match. Continue, until another user provides ABCDEF, too, (or, another user agree's e.g. with user's 2 response, it doesnt matter on what response they agree) and then assume the response ABCDEF to be “verified”. With our RCT, we implemented a treatment that modifies the environment in such a way that the observations get more similar, so the speed of verification is increased. With the RCT, we assess the causal effect. But we also want to quantify what it would mean if we implement the treatment in real, and therefore, we want to mimic and simulate 1:1 the real process.