6
$\begingroup$

I'm reading OpenIntro Statistics by David Diez and he says that for running an inference on a one sample mean we need to validate that the observations are independent:

enter image description here I'm running an A/B test and I want to make inference on the sample mean, do I need to discard 90% of experiments to satisfy this condition?

For more context: The A/B test is running on a website and each experiment is a user that gets assigned one of the 2 website variants.

$\endgroup$
2
  • 4
    $\begingroup$ This is a truly interesting question that (I hope) would shock the authors of that text (who are well known teachers), because I'm sure they never imagined this passage would be interpreted this way--but IMHO it's entirely their fault, because it's so vague and unhelpful. (I found it in the text at openintro.org/download.php?file=os3_tablet on p. 223 and verified that the context provides no clarification.) It gives me insight into how my own writing might be misinterpreted, which is humbling. $\endgroup$
    – whuber
    Jun 14, 2018 at 13:38
  • 2
    $\begingroup$ Ok I guess such a humbling experiencde can only come with a very wrong interpretation. I'll go with not discarding them. While still waiting for an explanation of that paragraph if somone can. $\endgroup$ Jun 14, 2018 at 14:49

1 Answer 1

5
$\begingroup$

You definitely do not need to discard 90% of your observations. The passage talks about sampling from a (finite) population. If your population had 10,000 units in it, the passage recommends you draw a sample of size less than 1,000. My intuition on the reason for this is doing so yields properties of the random sample that are similar to as if you were drawing from an infinite sample of independent observations (or drawing with replacement from a finite population). If your sample is a larger percentage of the population, dependence among the observations might be induced in the following way:

Imagine you had a population of 5 units and are sampling without replacement. If you've drawn two units randomly and are preparing to draw your third unit, the next draw depends on which of the other two units you selected; it is not independent of the other two. If you knew about your population and knew about who you already drawn, you can predict characteristics of who you draw next based on who you have drawn before. This is an independence violation.

Many of our statistical methods depend on drawing from an infinite population or drawing with replacement from a finite population; drawing without replacement from a finite population induces the dependence I described above. It would appear that drawing a small enough sample (i.e., 10% of the population) without replacement would approximate drawing a sample with replacement from the same population in terms of its statistical properties. This is probably why the authors made this recommendation.

This recommendation (probably) does not apply to your case. If you are "sampling" from a large enough population (i.e., all potential users of the website), then you will surely draw fewer than 10% of that population. The data you have collected in your sample should not suffer from a violation of independence due to the problem I described; if there is an independence violation, it is more related to the second clause in the passage (i.e., due to the design of your study).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.