4
$\begingroup$

What if, before you begin the data collection for an experiment, you randomly divide your subject pool into two (or more) groups. Before implementing the experimental manipulation you notice the groups are clearly different on one or more variables of potential import. For example, the two (or more) groups have different proportions of subjects by gender or age or educational level, or job experience, etc. What is a reasonable course of action in such a situation? What are the dangers of discarding the original random division of the subject pool and dividing the pool again? For example, are the inferential statistics that you might calculate based on the second set of groups in any way inappropriate due to the discarded first set of groups? For example, if we subscribe to discarding the first division of the subject pool into groups, are we changing the sampling distribution that our statistical test is based on? If so, are we making it easier or harder to find statistical significance? Are the possible dangers involved in repeating the division of subjects greater than the obvious danger of confounding due to group differences in educational level, say?

To make this question more concrete, assume for the sake of this discussion that the topic of the research is teaching method (and we have two teaching methods) and the difference noted between the two groups of subjects is level of formal education, with one group containing proportionally more people with highest educational attainment of high school level or less and the other group containing more people with some college or a college degree. Assume that we are training military recruits in a job that does not exist in the civilian world, so everyone entering that specialty has to learn the job from scratch. Assume, further, that the between group imbalance in previous educational attainment is statistically significant.

Parenthetically note that this question is similar to What if your random sample is clearly not representative?. In a comment there, @stask perceptively noticed that I am a researcher not a surveyor and commented that I might have gotten more relevant answers had I tagged my question differently, including "experiment design" rather than "sampling." (It seems the sampling tag attracts people working with surveys rather than experiments). So the above is basically the same question, in an experimental context.

Improve this question
$\endgroup$

4 Answers 4

Reset to default
3
$\begingroup$

If you just do a new randomization of similar type to the previous one (and allow yourself to keep randomizing until you like the balance) then it can be argued that the randomization is not really random.

However, if you are concerned about the lack of balance in the 1st randomization, then you probably should not be doing a completely randomized design in the first place. A randomized block or matched pairs design would make more sense. 1st divide the subjects into similar groups based on the things that you are most concerned about, prior education in your example, then do a randomization within each group/block. You will need to use a different analysis technique (randomized block, or mixed effects models instead of one-way anova or t-tests). If you cannot block on everything of interest then you should use the other techniques of adjusting for covariates that you do not block on.

Improve this answer
$\endgroup$
8
  • $\begingroup$ Yes, clearly you can argue that the second randomization is not fully random. Blocking is a good idea, but what if that was not done and the two independent groups formed just look uneven. Are there any ways to think about and quantify the effects of doing the formation of groups a second time? Is tossing the original groups out and doing a randomized blocks design better than just forming two independent groups a second time? If so, why/how? $\endgroup$
    – Joel W.
    Commented Aug 20, 2012 at 16:25
  • $\begingroup$ If the imbalance is enough to concern you then that says that you should have blocked to begin with. Most statistical techniques assume that your data is iid or at least exchangeable, if it is possible to get an imbalance large enough to concern you then that says your data is not iid or exchangeable. Proper blocking will give you conditional iid or conditional exchangeability (and the randomized block analysis is fine with the conditional). Proper blocking will also reduce variation and give you more power and better estimates of effect sizes. $\endgroup$
    – Greg Snow
    Commented Aug 20, 2012 at 16:42
  • $\begingroup$ In some fields, like psychology, there are so many potentially relevant variables that it may be impractical or even impossible to block on all the variables. That is why measurements in the social sciences have so much error variance. $\endgroup$
    – Joel W.
    Commented Aug 20, 2012 at 16:59
  • 2
    $\begingroup$ So pick a few. Some is better than nothing. $\endgroup$
    – Michael R. Chernick
    Commented Aug 20, 2012 at 19:35
  • 1
    $\begingroup$ Then measure them and look to see if they need to be adjusted for. $\endgroup$
    – Greg Snow
    Commented Aug 21, 2012 at 16:11
2
$\begingroup$

I do not think sampling should be adjusted because of chance imbalances. Adjusting creates complications that can be worse than any problem you think you might solve. If in the end you have covariate imbalances there are ways to adjust for them. See this book by Vance Berger for example.

Selection Bias and Covariate Imbalances In Randomized Clinical Trials

Improve this answer
$\endgroup$
6
  • $\begingroup$ A quick look at the Amazon preview of the interesting Berger text you cite suggests that treatment effects may be due to selection bias and, therefore, claims for treatment superiority can be challenged if there is selection bias (e.g., page 39, paragraph 3). What part of the text were you referring to when you said "complications that can be worse than any problem you think you might solve"? $\endgroup$
    – Joel W.
    Commented Aug 20, 2012 at 12:50
  • $\begingroup$ Assume that I conclude that the educational method used with the higher educational attainment group is more effective than the educational method used with the other group. How do I respond to a (non-statistician) manager who says, could not the apparent superiority of the educational method used with the higher educational attainment group be due instead to the higher educational attainment of the people in the group or correlates of that, such as the people in that group being brighter? $\endgroup$
    – Joel W.
    Commented Aug 20, 2012 at 13:02
  • $\begingroup$ What I was suggesting is that Berger provides methods for making adjustments for covariate imbalance. The other comment wasn't referring to Berger's text. $\endgroup$
    – Michael R. Chernick
    Commented Aug 20, 2012 at 13:43
  • $\begingroup$ You say, "Adjusting creates complications that can be worse than any problem you think you might solve." But I am asking a different question: whether to toss out the first random division of the subject pool into groups in favor of a second such division. What possible complications might that second division lead to? $\endgroup$
    – Joel W.
    Commented Aug 20, 2012 at 13:50
  • $\begingroup$ @JoelW. If you start out with a random sample and manipulate it like that what you are left with is no longer random. So there could be bias in making inference that you can't quantify. $\endgroup$
    – Michael R. Chernick
    Commented Aug 20, 2012 at 14:00
1
$\begingroup$

In my opinion, drawing and redrawing a sample in response to not looking "random enough" doesn't generate any bias so long as

  • You aren't making that decision based upon your outcome of interest and
  • You condition on your observables.

We know that bias is introduced by selection on unobservables. As long as you perform selection based upon observables and those observables are conditioned on in the analysis, you're okay.

Is this still random? I say yes, it is a random sample, conditional on the observables. This is what matters. We need to be careful about how we are defining "random" and when we are precise, we see that randomization is still here.

How might you condition on them? Linear models are a standard method. Matching is a nice non-parametric procedure, but requires a bit more thought that you might imagine (I'm biased to be fond of genetic matching algorithms).

I would note that the conditioning process smooths out or takes into account (depending upon how you want to look at it) the imbalances present. So you don't need to redraw your sample. The only time that you might need to do this is if you want to do matching and you don't have common support between the two groups (you have some college graduates in one group and none in the other, for example).

Blocking or stratifying is really about variance reduction (unless your interested in non-homogeneous responses along some dimension), not about bias.

Since you still have randomization, no adjustments need to be made in your testing procedures. (Though, if you're doing matching, be sure to use variances estimators designed for matching; see Abadie and Imbens (2006))

Improve this answer
$\endgroup$
0
$\begingroup$

An experiment requires a control, but it doesn't need to be structured as "end result of one randomly assigned group" vs. "end result of another randomly assigned group". You are concerned that the composition of the groups is unbalanced on one particular attribute that you believe has an outsized influence on the end result; perhaps a different structure would make this irrelevant. The experiment may be set up as a before-and-after for each teaching method group, or it can be a regression where one independent variable is the teaching method. If you have really strong reasons to believe that this one particular attribute outweighs all others, you could even take your population of participants and separate them into groups according to education level, and then within each group randomly assign participants to the control and test groups.

A risk is that while you've picked out one key driver of different outcomes, there may be others lurking in there you haven't detected, and which you might be subverting.

Improve this answer
$\endgroup$
9
  • $\begingroup$ In some fields, like psychology, there are so many potentially relevant variables that it may be impractical or even impossible to block on all the variables. But it may be that we notice a troubling difference after we randomly assign subjects to groups. That is the situation I ask about. $\endgroup$
    – Joel W.
    Commented Aug 21, 2012 at 18:18
  • $\begingroup$ Agreed - that is my caveat in the last paragraph. There are some situations in which you have prior knowledge about which variables have the most impact on the final outcome, but it sounds like you're saying that in this hypothetical, you do not. That would affect the "blocking" approach, but not a different method of comparing test to control, such as before-and-after, or through regression. $\endgroup$
    – Jonathan
    Commented Aug 23, 2012 at 5:18
  • $\begingroup$ So there are two possible risks: (1) there "may be others lurking" versus (2) a clear, unintended difference between independent experimental groups. My question is, how to compare the risks. It seems to me that the risk of using clearly different groups in an experiment is greater than the possibility of a potential lurking risk. My question is, is it possible to compare such risks in a quantitative fashion, or at least clearly state the risks in a qualitative fashion? (The lurking risk you mention is a rudimentary step in that direction.) $\endgroup$
    – Joel W.
    Commented Aug 23, 2012 at 12:29
  • $\begingroup$ Are we assuming that the test needs to be a strict comparison of one final metric of the control group, compared to one final metric of the test group? While I appreciate the theoretical value of your question, in practice we may be able to make the problem go away with a different method of comparison :) $\endgroup$
    – Jonathan
    Commented Aug 23, 2012 at 21:33
  • $\begingroup$ What do you mean by "a different method of comparison"? In the question posed, I envisioned two groups, trained with two different teaching methods, evaluated after training with the same achievement test designed to measure mastery of the material. The performance of the two groups on this achievement test would be compared, perhaps with a t test. $\endgroup$
    – Joel W.
    Commented Aug 23, 2012 at 22:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.