1
$\begingroup$

Disclaimer: I'm not sure if there's a better stackexchange for this, but since I consider experimental design an integral part of causal/statistical inference, I'm hoping this is an appropriate audience.

So, I'm considering setting up a controlled experiment on a service that has users (each have unique user_ids). To determine which users get put into which group ("control" or "treatment"), an engineer has proposed the following assignment algorithm:

if (md5(user_id) mod 100 < control_fraction)
    assign user_id to control
else
    assign user_id to treatment

Now, I've empirically determined that for $X \in \mathbb{N}$, $$ md5(X) \mod 100 \sim \mathcal{U}nif[0,100) $$ Here's the distribution of values when $X \in \{1,2,\ldots,100k\}$ enter image description here

But I'm still left with lingering doubts as to whether or not this assignment mechanism may introduce some sort of subtle selection/enrollment biases...

Does this scheme allow me to proceed with my experimental analysis, as if we had assigned users randomly, or do I need to take anything into account?

$\endgroup$
  • $\begingroup$ Any random number generated by a computer is only a deterministic algorithm that _behaves quite like_a random variable. This one looks good enough to me (but I am too new to statistics to formulate it as an answer). $\endgroup$ – Dirk Horsten Aug 4 '15 at 4:33
3
$\begingroup$

This question has been sitting here a while, but better late than never.

There is a very slight bias with the given approach. Applying the md5 algorithm will result in a uniform distribution in the 128 bit output space. However, that means there are $2^{128}$ possible outputs. $2^{128} \equiv 56\space (\text{mod } 100)$, so you're slightly more likely to get users in the 0-56 range and less likely to get users in the 57-99 range than you should be.

At the end of the day, this only means that more users could end up in the control group than expected. (Although assuming you have at most 7.3 billion users, this issue is very unlikely to manifest.)

Is there a reason you can't simply use a random number instead of modding an md5?

$\endgroup$
  • 1
    $\begingroup$ Welcome to our site - note that you can use Latex formatting here by putting text between dollar signs, e.g. $2^{128}$ produces $2^{128}$ $\endgroup$ – Silverfish Sep 13 '16 at 21:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.