7
$\begingroup$

I have been using the term selection bias to refer to a situation where (e.g.) schools with certain pre-existing characteristics are more likely to be included in (e.g.) a teacher training programme than others, and these background characteristics may affect their results, making it harder to assess whether the intervention has had an impact or not.

I now see (e.g. on Wikipedia) that it is most often used to refer to differential probability of selection to the study sample rather than differential probability of selection to the programme treatment group.

Can it be used correctly in either case and if not, what is the correct term for bias caused by some types of unit being more likely to be treated than others? Or should this be called "confounding" rather than bias?

Improve this question
$\endgroup$
4
  • $\begingroup$ The second sentence under "Avoidance" at that article you link to seems to suggest it encompasses that case. [Personally, I'd use it that way, since it relates to the stage of 'selection' into treatment-group] $\endgroup$
    – Glen_b
    Commented Jan 26, 2015 at 22:18
  • $\begingroup$ @Glen_b You mean "An informal assessment of the degree of selection bias can be made by examining correlations between exogenous (background) variables and a treatment indicator."? I'm not sure I see how that sheds light on the question - can you explain? $\endgroup$
    – Stuart
    Commented Jan 27, 2015 at 1:44
  • $\begingroup$ If the treatment indicator is correlated with background variables, assignment to treatment was not "at random" (i.e. with equal probability across background variables). Since that apparently counts as selection bias, that seems to speak directly to your question. $\endgroup$
    – Glen_b
    Commented Jan 27, 2015 at 1:47
  • $\begingroup$ @Glen_b But examining the correlation of the exogenous variables and treatment indicator within the sample would also allow you to examine selection bias to the sample, assuming equal assignment to the treatment group in the population, so I'm not sure this advice says much about what counts as selection bias. $\endgroup$
    – Stuart
    Commented Jan 27, 2015 at 1:50

3 Answers 3

Reset to default
5
$\begingroup$

I think these two are actually very similar. In the experimental settings, you are comparing some mean outcome for treated versus control and you are worried that there might be individuals with some observed characteristics who appear only among participants or non-participants (or appear more often) and/or that this is happening with some unobserved characteristics, which is a much harder problem.

In the prototypical survey setting, you are essentially comparing wages for low-education (C) and high-education (T) women, and you are worried that you are not observing women with large negative unobservables in the low education groups because they are not in the labor force. Education might be multivalued rather than binary, but the spirit of this comparative exercise is the same.

In both cases, you are using the average outcome for the control or low-education groups in place of what would have happened to the treated or high-education group in the absence of that treatment or education, which is something you don't see.

Improve this answer
$\endgroup$
2
$\begingroup$

I like @DimitriyV.Masterov's answer (+1); they are very similar and you could probably use "selection bias" for "differential probability of selection to the programme treatment group". However, I am somewhat uncomfortable with that usage and think it would probably be better to use different phrasing.

You don't really select people into treatment groups, you assign them. If the probability of a person being assigned to a treatment group is not independent of their attributes (e.g., healthier patients are more likely to go into the control group), then I think it is better to say that the assignment is confounded.

On the other hand, if your study is observational in nature, there is no assignment at all. The status of all variables, whether categorical (sick / healthy) or continuous (weight), should be understood as endogenous / correlated with unknown confounders. In the world as we find it (that is, without our acting on the world exogenously by manipulating the levels of a variable and assigning people to those levels), everything is ultimately related to everything somehow. It may well be that in selecting your sample, you were more likely to draw people with certain properties than people with different properties, such that the (say) proportion of people in your sample dieting to lose weight is higher than in the population (and the proportion of people exercising to lose weight is lower that in the population). But this is not assignment. I would call that situation a biased "selection to the study sample", but I wouldn't call it a biased "selection to the programme treatment group".

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ While I think you are 100% correct, in most practical cases I have experienced (I work in a research shop in a federal agency where we run tons of program evaluations) we often refer to both situations simply as selection bias. Depending on the technical level of the OP's audience, in my experience people use selection bias for both cases. $\endgroup$
    – robin.datadrivers
    Commented Jan 29, 2015 at 2:50
0
$\begingroup$

Selection Bias occurs when there is no appropriate randomization achieved while selecting individuals, groups or data to be analysed.

Selection bias implies that the obtained sample does not exactly represent the population that was actually intended to be analyzed.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.