2
$\begingroup$

Is there a mathematical difference between randomly selecting a population of users before an event vs after an event, for the purposes of experimentation?

For example, let's say we want to split all users that perform event X into two groups, A and B. A is 90% of the users (randomly selected) and B is 10% of the users.

Is there any difference if we first randomly split all users into two groups, A and B, at 90% and 10% respectively, and then of all those users, those who perform event X, they will then be sorted based on their already-membership in group A or B?

$\endgroup$
4
  • 1
    $\begingroup$ Your example is unclear. You have event A and group A. So when you say A is 90% of the users, which A is being referred to. Btw, why would you use the same letter to describe two separate entities in an example? $\endgroup$
    – Pierre L
    Jul 12, 2016 at 20:25
  • $\begingroup$ sorry about that, I didn't proofread. I initially wrote 90%/10% but then thought to simplify it with A and B, but then forgot I used A for the event name. $\endgroup$
    – Ste
    Jul 12, 2016 at 20:31
  • 1
    $\begingroup$ The descriptions of "sorting" and "splitting" are obscure in this question. Could you provide a small example to show what it is you are contemplating? $\endgroup$
    – whuber
    Jul 12, 2016 at 20:59
  • $\begingroup$ @whuber, sorry that's my fault. I meant sorting and splitting to mean the same thing but I wasn't sure exactly how it should be worded. In the above example, what I meant to say was: once a user performs event X, then afterwards, all the participants would then be put into the same A and B buckets that they were selected in before the event X. $\endgroup$
    – Ste
    Jul 12, 2016 at 22:02

1 Answer 1

3
$\begingroup$

I will answer with a cautious No.

Imagine that I select 100 volunteers to play a carnival game. They must make a basketball shot. That is our event X. Let's say we know that the average person makes the shot 60% of the time.

Scenario 1 - Divide first then shoot

90 of the volunteers go to group A and 10 are assigned to group B. 60% of the volunteers make the shot as expected. Leaving us with 54 (90*0.6) in A and 6 (10*0.6) in B.

Scenario 2 - Shoot first then divide

All 100 volunteers shoot the basket and 60% make the shot as expected. Group A will have 54 and B will have 6.

Conclusion

No difference. The same probability distribution results. A smarter person will be able to prove this mathematically. But the crucial bit is that membership to a group must, in itself, not alter the probability of event X occurring.

$\endgroup$
4
  • $\begingroup$ Thank you. So if there is any bias in terms of how event X were achieved, for example, basketball players would be naturally selected for, then that would alter the results and the grouping would be different? $\endgroup$
    – Ste
    Jul 12, 2016 at 22:12
  • $\begingroup$ Yes. If there is something different about the groups that will affect the outcome then you group prior to the event. $\endgroup$
    – Pierre L
    Jul 12, 2016 at 22:14
  • $\begingroup$ The two are slightly different in the possibly irrelevant sense that in Scenario 2, exactly 90 percent will be assigned Group A by construction while in Scenario 1, on average 90 percent will be assigned Group A. But the two are basically the same in the sense that if assignment to Group A is random, it will be independent of event $X$ and all other predetermined unobservables in either scenario (unless of course event $X$ is somehow a function of assignment to group A). $\endgroup$ Jul 13, 2016 at 0:14
  • $\begingroup$ As Pierre mentioned, "the crucial bit is that membership to a group must, in itself, not alter the probability of event X occurring." For example, if Group A eats a big tub of ice cream, they may get a belly ache and miss free throws at a higher rate (i.e. event X is less likely to occur if assignment to Group A occurs first). $\endgroup$ Jul 13, 2016 at 0:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.