# What happens if I try to match on outcome probability? (rather than propensity score/treatment probability)

I am applying propensity score matching in some observational data analysis to determine whether a certain treatment has significant effect on an outcome, after controlling for some covariates.

I realize I don't really know the answer to the question "why not use outcome probabilities?" That is, why not precisely the same procedure, except replacing treatment probability with outcome probability?

With "outcome probability matching," intuitively your matched pairs will be pairs of patients who are similarly at risk of getting the outcome, and you'll be checking whether the treatment makes it less likely.

Is there something fundamentally unsound with this?

[Note I don't think I really understand propensity scores -- it just seems to be one reasonable way to account for covariates; perhaps it is canonical in some sense/etc.]

I notice the term "prognostic score" in the comments in this question Do propensity scores reflect the probability of treatment or outcome? I will read up on this, but asking anyway I guess for references or original perspectives on like why matching on outcome probabilities doesn't make sense.

## 1 Answer

The "probability of outcome" is indeed called the prognostic score. I recommend Hansen (2008) for a full account of prognostic scores and their value in causal inference. Like propensity scores, they are balancing scores: within strata of the prognostic score, covariates are balanced and confounding is eliminated. This means you can match or stratify on the prognostic score to arrive at unbiased estimates of a treatment effect, as you propose.

The prognostic score can be used in balance assessment after using propensity scores, as described by Stuart et al. (2013). Using the prognostic score for balance assessment yielded superior performance over traditional methods, such as comparing covariate means or cumulative density functions. To implement balance assessment with prognostic scores in R, check out the main vignette for the cobalt package.

There are a few disadvantages to using prognostic scores. First, using them fails to separate the design and analysis phases of a study. In the design phase, the outcome is not considered, and work is done to eliminate sources of confounding (e.g., through quasi-experimental design or propensity score pre-processing); in the analysis phase, the outcome is considered and the treatment effect is estimated. Several authors have warned about the dangers of blurring the two phases: Rubin (2001) and King & Nielsen (2019) describe the problems of researchers tailoring their models for specific desired results, and Hansen (2008) notes that using the prognostic score to adjust for confounding in the same sample from which it was estimated leads to an increased type I error rate. For many causal inference researchers, retaining this separation is important for maintaining the validity and trustworthiness of their inferences, and prognostic scores violate this principle.

Another problem with prognostic scores is that in general, outcome model-based approach to estimating treatment effect outperform treatment model-based approaches because the reduction in variance of the former often outweighs the reduction in bias of the latter. Advanced techniques like targeted maximum likelihood estimation and Bayesian additive regression trees (which I describe briefly in this post) have proven to be extremely effective at estimating treatment effects (see Dorie et al., 2019, for example), and both rely on modeling both the treatment and the outcome and using both models to reduce bias in a treatment effect estimate, often in a single step without the treatment model validation process that typically accompanies propensity score analysis. The key is, if you're going to violate the principle of separating the design and analysis phases and model the outcome anyway (i.e., in order to use prognostic scores), you might as well just use the outcome model to estimate a treatment effect.

Perhaps because of these issues (desire to maintain the distinction between design and analysis and ability to use better methods if the distinction is to be broken), prognostic score methods have not caught on. Although Hansen (2008) has over 250 citations (at the time of writing based on Google Scholar), most of them are methodological papers on propensity scores that simply mention prognostic scores. The methodological literature has reached a point where even propensity scores (at least used alone as originally described by Rosenbaum & Rubin, 1983) are being seen as obsolete as robust machine learning and optimization-based approach are becoming popular.

Dorie, V., Hill, J., Shalit, U., Scott, M., & Cervone, D. (2019). Automated versus do-it-yourself methods for causal inference: Lessons learned from a data analysis competition. Statistical Science, 34(1), 43–68. https://doi.org/10.1214/18-STS667

Hansen, B. B. (2008). The prognostic analogue of the propensity score. Biometrika, 95(2), 481–488. https://doi.org/10.1093/biomet/asn004

King, G., & Nielsen, R. (2019). Why Propensity Scores Should Not Be Used for Matching. Political Analysis, 1–20. https://doi.org/10.1017/pan.2019.11

Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55. https://doi.org/10.1093/biomet/70.1.41

Rubin, D. B. (2001). Using Propensity Scores to Help Design Observational Studies: Application to the Tobacco Litigation. Health Services and Outcomes Research Methodology, 2(3–4), 169–188. https://doi.org/10.1023/A:1020363010465

Stuart, E. A., Lee, B. K., & Leacy, F. P. (2013). Prognostic score-based balance measures can be a useful diagnostic for propensity score methods in comparative effectiveness research. Journal of Clinical Epidemiology, 66(8), S84. https://doi.org/10.1016/j.jclinepi.2013.01.013