I was wondering if someone has ideas to statistically examine selection into programme participation? For example, would it make sense to present the results of a propensity score analysis?
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.Sign up to join this community
By "selection bias," I assume you mean "confounding", i.e., that there are common causes of treatment assignment and variation in the outcome. It is important to distinguish between two things here: 1) describing how participants and non-participants differ with respect to potential confounding variables and 2) building a causal model that describes selection into program participation.
For goal 1), a balance table does exactly what you want. It is literally a table displaying how the participants and non-participants differ from each other. This is important to let readers know what the goal of an analysis is that seeks to remove confounding (e.g., propensity score or regression analysis). Readers can examine the balance table before and after adjustment and decide for themselves if enough variables have been accounted for and if balance is sufficient to warrant trust in the effect estimate. A balance table doesn't describe how participation is selected, but just describes the consequences of non-random selection into participation. That is, it describes the consequences of the treatment assignment mechanism. In terms of visual display, if you want to avoid a table, you can look into making a Love plot. The R package
cobalt makes these easily and has examples in its documentation.
For goal 2), a balance table is not sufficient, and, in fact, it will be vary hard to describe an accurate causal model for selection unless the process is very well understood. You might be tempted to run a logistic regression predicting participation from the covariates, but that doesn't mean the covariates cause selection into participation; the collected covariates may just be correlated with the true causes of participation and so will have nonzero coefficients even though they don't actually explain how participation is selected. Asking how participation is selected is a whole scientific question in itself that should be answered separately from estimating the causal effect of participation. Ideally, it should be answered before estimating the effect of participation so that confounding can be adequately address based on the newfound knowledge of the selection process. In practice, though, you don't need a solid causal model of selection if you have a large collection of potential confounding variables that can be adjusted for.