I was wondering if someone has ideas to statistically examine selection into programme participation? For example, would it make sense to present the results of a propensity score analysis?

Thank you


By "selection bias," I assume you mean "confounding", i.e., that there are common causes of treatment assignment and variation in the outcome. It is important to distinguish between two things here: 1) describing how participants and non-participants differ with respect to potential confounding variables and 2) building a causal model that describes selection into program participation.

For goal 1), a balance table does exactly what you want. It is literally a table displaying how the participants and non-participants differ from each other. This is important to let readers know what the goal of an analysis is that seeks to remove confounding (e.g., propensity score or regression analysis). Readers can examine the balance table before and after adjustment and decide for themselves if enough variables have been accounted for and if balance is sufficient to warrant trust in the effect estimate. A balance table doesn't describe how participation is selected, but just describes the consequences of non-random selection into participation. That is, it describes the consequences of the treatment assignment mechanism. In terms of visual display, if you want to avoid a table, you can look into making a Love plot. The R package cobalt makes these easily and has examples in its documentation.

For goal 2), a balance table is not sufficient, and, in fact, it will be vary hard to describe an accurate causal model for selection unless the process is very well understood. You might be tempted to run a logistic regression predicting participation from the covariates, but that doesn't mean the covariates cause selection into participation; the collected covariates may just be correlated with the true causes of participation and so will have nonzero coefficients even though they don't actually explain how participation is selected. Asking how participation is selected is a whole scientific question in itself that should be answered separately from estimating the causal effect of participation. Ideally, it should be answered before estimating the effect of participation so that confounding can be adequately address based on the newfound knowledge of the selection process. In practice, though, you don't need a solid causal model of selection if you have a large collection of potential confounding variables that can be adjusted for.

  • 1
    $\begingroup$ Thank you Noah for the excellent conceptual differentiation and warning against my idea of running a logistic model to predict the propensity scores. In case you have experience with modelling selection, I would be very interested to hear more from you or citations to literature. Unfortunately, this topic is difficult to research, because the key words always lead to models which try to remedy selection effects while answering causal question of programme participation, not just the step of modelling selection. Or perhaps it is not a good idea at all to try to do it statistically? $\endgroup$
    – Papayapap
    Oct 29 '19 at 10:34
  • 2
    $\begingroup$ I don't, sorry. My expertise is in controlling for confounding, not understanding selection processes. I'm glad you found this distinction helpful. $\endgroup$
    – Noah
    Oct 29 '19 at 19:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.