9
$\begingroup$

Random assignment is valuable because it ensures independence of treatment from potential outcomes. That is how it leads to unbiased estimates of the average treatment effect. But other assignment schemes can also systematically ensure independence of treatment from potential outcomes. So why do we need random assignment? Put another way, what is the advantage of random assignment over nonrandom assignment schemes that also lead to unbiased inference?

Let $\mathbf{Z}$ be a vector of treatment assignments in which each element is 0 (unit not assigned to treatment) or 1 (unit assigned to treatment). In a JASA article, Angrist, Imbens, and Rubin (1996, 446-47) say that treatment assignment $Z_i$ is random if $\Pr(\mathbf{Z} = \mathbf{c}) = \Pr(\mathbf{Z} = \mathbf{c'})$ for all $\mathbf{c}$ and $\mathbf{c'}$ such that $\iota^T\mathbf{c} = \iota^T\mathbf{c'}$, where $\iota$ is a column vector with all elements equal to 1.

In words, the claim is that assignment $Z_i$ is random if any vector of assignments that includes $m$ assignments to treatment is as likely as any other vector that includes $m$ assignments to treatment.

But, to ensure independence of potential outcomes from treatment assignment, it suffices to ensure that each unit in the study has equal probability of assignment to treatment. And that can easily occur even if most treatment assignment vectors have zero probability of being selected. That is, it can occur even under nonrandom assignment.

Here is an example. We want to run an experiment with four units in which exactly two are treated. There are six possible assignment vectors:

  1. 1100
  2. 1010
  3. 1001
  4. 0110
  5. 0101
  6. 0011

where the first digit in each number indicates whether the first unit was treated, the second digit indicates whether the second unit was treated, and so on.

Suppose that we run an experiment in which we exclude the possibility of assignment vectors 3 and 4, but in which each of the other vectors has equal (25%) chance of being chosen. This scheme is not random assignment in the AIR sense. But in expectation, it leads to an unbiased estimate of the average treatment effect. And that is no accident. Any assignment scheme that gives subjects equal probability of assignment to treatment will permit unbiased estimation of the ATE.

So: why do we need random assignment in the AIR sense? My argument is rooted in randomization inference; if one thinks instead in terms of model-based inference, does the AIR definition seem more defensible?

Improve this question
$\endgroup$
8
  • 3
    $\begingroup$ I haven't read Angrist et al., so maybe I'm missing something, but I have a quibble w/ your phrasing. We don't use random assignment to insure that the treatment is independent of the potential outcomes. Whether the treatment is independent of the outcomes in a true experiment depends on whether there is a direct causal connection b/t the treatment & the outcome. Rather, random assignment insures that the treatment is independent of lurking variables (or, potential confounders). It is the possibility that the outcome was caused by something other than the treatment that we hope to exclude. $\endgroup$ – gung - Reinstate Monica Sep 22 '12 at 1:38
  • 1
    $\begingroup$ @gung, I think that you are conflating "potential outcomes" and "outcomes." It's true that random assignment doesn't ensure independence of treatment from outcomes (that is, from observed outcomes). But potential outcomes are not the same as observed outcomes, and random assignment does ensure independence of treatment from potential outcomes. I won't edit the original post to expand on this point; doing so would take me too far afield from the main topic. But en.wikipedia.org/wiki/Rubin_causal_model may be helpful on this point. $\endgroup$ – user697473 Sep 22 '12 at 1:46
  • 3
    $\begingroup$ "[T]o ensure independence of potential outcomes from treatment assignment, it suffices to ensure that each unit in the study has equal probability of assignment to treatment." This is incorrect. Suppose you have enrolled $x$ males and $x$ females in a study. Flip a fair coin: if heads, assign all females to the treatment group (and all males to the control group); if tails, all males will be in the treatment group and all females in the control group. Each subject (obviously) has 50% chance of assignment to the treatment group--but treatment is completely confounded with gender. $\endgroup$ – whuber♦ Sep 22 '12 at 16:27
  • 1
    $\begingroup$ @whuber, your comment doesn't sound correct. To see why, suppose $x$ = 1. The man's potential outcomes are Y(1) = 1 and Y(0) = 0. (That is, $Y_m$ = 1 if the man is treated, 0 if not.) For the woman, the potential outcomes are Y(1) = -1 and Y(0) = 2. (The particular potential outcomes don't matter much, but small integers keep things simple.) Then E[Y(1) | Z] = E[Y(1)] = 0. Similar equalities hold for E[Y(0)]. More generally, your assignment mechanism isn't confounded with gender, and it will produce an unbiased ATE estimate. If I am misunderstanding something, please let me know. $\endgroup$ – user697473 Sep 22 '12 at 18:17
  • 3
    $\begingroup$ Sure, the estimate is "unbiased" in the same sense that a stopped clock gives an unbiased estimate of the time! Actually, it's worse than that: this method of random selection yields results that cannot be attributed to the treatment, because they can just as well be attributed to gender. That's what confounding means. Focusing on obtaining unbiased results while destroying all useful information in the experiment is the proverbial throwing out of the baby... $\endgroup$ – whuber♦ Sep 22 '12 at 18:49
8
$\begingroup$

This follows up on gung's comment. Overall average treatment effect is not the point.

Suppose you have $1000$ new diabetes cases where the subject is between the ages of $5$ and $15$, and $1000$ new diabetes patients over $30$. You want to assign half to treatment. Why not flip a coin, and on heads, treat all of the young patients, and on tails, treat all of the older patients? Each would have a $50\%$ chance to be selected fro treatment, so this would not bias the average result of the treatment, but it would throw away a lot of information. It would not be a surprise if juvenile diabetes or younger patients turned out to respond much better or worse than older patients with either type II or gestational diabetes. The observed treatment effect might be unbiased but, for example, it would have a much larger standard deviation than would occur through random assignment, and despite the large sample you would not be able to say much. If you use random assignment, then with high probability about $500$ cases in each age group would get the treatment, so you would be able to compare treatment with no treatment within each age group.

You may be able to do better than to use random assignment. If you notice a factor you think might affect the response to treatment, you might want to ensure that subjects with that attribute are split more evenly than would occur through random assignment. Random assignment lets you do reasonably well with all factors simultaneously, so that you can analyze many possible patterns afterwards.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you, Douglas. This answer makes sense to me. For the record, I didn't have in mind anything as extreme as your example or @whuber's example above. I was thinking instead of cases in which we eliminate from consideration just a few treatment vectors. (Consider a case in which a client says "you can treat this person or that one, but not both.") But I think that your general points hold even for the milder cases that I have in mind. $\endgroup$ – user697473 Sep 26 '12 at 13:05
  • $\begingroup$ I think if you only eliminate a few vectors, then you don't change the amount of information you can extract by much. Quantifying this accurately may be messy -- there are naive bounds which are probably too pessimistic. $\endgroup$ – Douglas Zare Sep 26 '12 at 17:55
  • $\begingroup$ @DouglasZare I have a question about your extreme example. I believe the goal is to find whether the treatment is effective for the population which has both young and old patients. Then, your method will generate two samples that cannot be regarded as the representative sample from the potential outcome distribution $F_t$ where all people take treatment and potential outcome distribution $F_c$ where all people take control. So then your observed treatment effect is biased $\endgroup$ – KevinKim Dec 10 '16 at 5:04
1
$\begingroup$

In your example you can leave 2 and 5 out as well and not contradict yourself. At an item level there's still an equal chance of being 1 or 0 when there's just a 1:1 odds of selecting 1 or 6. But, now what you did by removing 3 and 4 becomes more obvious.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks, John. Yes, you are correct. It seems that we can eliminate as many treatment assignment vectors as we like, in any combination, so long as we use the remaining vectors in a way that gives each unit equal probability of assignment to treatment. $\endgroup$ – user697473 Sep 22 '12 at 18:14
  • $\begingroup$ I don't think you're getting what I'm saying. What I've presented is the ad absurdum case for your argument that argues against it. $\endgroup$ – John Sep 22 '12 at 18:52
  • $\begingroup$ Your example is extreme, but I don't see anything absurd about it. It's a valid demonstration of the point: nonrandom assignment schemes (like using only vectors 1 and 6) can lead directly to unbiased estimation of the average treatment effect. It follows that we don't need random assignment to get unbiased ATE estimates. Of course, there may yet be reasons why it's bad to eliminate vectors 2 through 5. (See Douglas Zare's comment above.) I haven't yet thought through these reasons. $\endgroup$ – user697473 Sep 22 '12 at 19:47
  • $\begingroup$ You should. It's why you can't eliminate them. $\endgroup$ – John Oct 5 '12 at 21:40
1
$\begingroup$

Here's another one of the lurking or confounding variables: time (or instrumental drift, effects of sample storage, etc.).
So there are arguments against randomization (as Douglas says: you may do better than randomization). E.g. you can know beforehand that you want your cases to be balanced over time. Just as you can know beforehand that you want to have gender and age balanced.

In other words, if you want to manually choose one of your 6 schemes, I'd say that 1100 (or 0011) is a decidedly bad choice. Note that the first possibilites you threw out are those that are most balanced in time... And the worst two are left in after John proposed to thow out also 2 and 5 (against which you did not protest).
In other words, your intuition which schemes are "nice" unfortunately leads towards bad experimental design (IMHO this is quite common; maybe ordered things look nicer - and for sure it is easier to keep track of logical sequences during the experiment).

You may be able to do better with non-randomized schemes, but you are also able to do much worse. IMHO, you should be able to give physical/chemical/biological/medical/... arguments for the particular non-random scheme you use, if you go for a non-random scheme.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.