# balance in propensity score

I often hear many authors talk about how propensity score helps achieving balance or similarity between treatment groups. Propensity score collapsing information about all the matching variables into a single summary statistics that measures how much an observation resembles treatment observations on baseline characteristics etc...

I am not sure I understand this clearly. What does this mean ? "Propensity score collapsing information about all the matching variables into a single summary statistics"

What about the effect of covariates ? How is this estimated if Propensity score collapses information about all the matching variables into a single summary statistics.

Please pardon my ignorance. Need some help understanding this with some examples. Thanks in advance.

One way of estimating causal effects is to regress the outcome on the treatment and covariates. You cannot interpret the coefficients on the covariates in this type of model as the "effect of covariates". That is known as the table 2 fallacy. You can say nothing about the relationship between covariates and the outcome based on an outcome model focused on removing confounding from a treatment by covariates. So whether you are using propensity scores or not, you cannot talk about the "effect of covariates".

Propensity scores are one way of isolating the relationship between the treatment and the outcome by removing the association between the treatment and the covariates. Rather than having to ensure your model regressing the outcome on the treatment and covariates is correctly specified (it isn't), you can use propensity score methods, which involve collapsing the covariate information into a single variable, and then conditioning on the propensity score instead of the covariates. There are several ways of doing this, including matching and weighting, which serve to adjust the distribution of covariates in the sample so that the treatment is independent of the covariates. So, given that you can't interpret the effects of covariates in an outcome model when using covariate adjustment, you aren't losing that information by using propensity scores instead, because you didn't have it to begin with.

Most propensity score methods allow you to estimate marginal effects rather than conditional effects. When your outcome is binary or time-to-event and you are using a noncollapsible measure of effect like the odds ratio or hazard ratio, the conditional effect will not be equal to the marginal effect, so regression and propensity score methods do not even target the same quantity of interest. This is Frank's point. A marginal effect averages over any possible effect heterogeneity. Frank argues using propensity scores discards useful information and degrades statistical performance relative to a well-specified outcome regression model, rendering propensity scores unhelpful in all but a few corner cases (e.g., when there are too few events per covariate).

To learn more about propensity score analysis, I suggest you read one of the many excellent articles out there. In particular, I recommend Austin (2011). And always remember that propensity score analysis is an advanced statistical method that requires extensive training to use correctly.

• this is an excellent explanation. I will ask you some followup questions after reading that article from Peter Austin (2011) Nov 4, 2021 at 2:48
• Noah is it possible dissimilar subjects in the treatment & control groups could share the same propensity score and get matched despite being very different? Like males < 20 yrs old in the trmt group getting matched to females > 60 yrs old in the control group? Would mismatches like this always be spotted when doing the post-matching balance check? Mar 16, 2022 at 1:54
• @RobertF this would make a great separate question. Short answer is yes. This is precisely the motivation behind King & Nielsen (2019). Balance is a sample quality, so the closeness of individuals doesn't matter. As long as the distributions are similar, you have balance. Close pairs can improve balance but the whole point of propensity scores is to have balance without close pairs (i.e., closeness on the covariates).
– Noah
Mar 16, 2022 at 3:46
• So if we're assessing balance by only comparing covariate means in treatment & control groups, perfect or closely balanced groups can conceal mismatches. Which we avoid when using methods like inverse probability weighting or g-computation (although then we introduce other limitations). Are balance diagnostics a waste of time? Mar 16, 2022 at 13:56
• Quite the opposite; balance is all that matters, and having close pairs is incidental. It doesn't matter whether your pairs are close if your goal is to reduce bias due to confounding. If you discard pair membership after matching what you're left with is IPW so you might say IPW is even worse than matching at forming close pairs since no pairs are formed at all; IPW only produces balance in the overall sample. G-comp is totally different from the two but closer to weighting; it does no pairing.
– Noah
Mar 16, 2022 at 14:17

Typically propensity scores are modelled by taking pre-treatment covariates and including them in a model that predicts the probability of receiving the given treatment (most frequently this is done by fitting a logistic regression function that predicts treatment assignment to pre-treatment covariates, but other methods exist too like boosted regression and other machine learning methods). So in that sense, the predicted probability or propensity scores is a single number that is created from a function of the pre-treatment covariates ("all the matching variables," as you say). However, assessing balance is an entirely different endeavor. To assess balance, one must examine whether significant differences in the distributions of covariates still exist after adjusting for the propensity score.

You'll find this reference to be a quick read that describes covariate balance and propensity scores in greater detail: https://cran.r-project.org/web/packages/MatchIt/vignettes/assessing-balance.html

You have hit on some key questions. By ignoring covariates except in how they relate to treatment choice, you are not accounting for outcome heterogeneity when doing propensity score analysis (if done the usual way). This results in a great loss of power and biased estimation of treatment effects due to non-collapsibility of odds ratios etc. Propensity scores are needed (for the pure confounding part of the analysis, not for the outcome heterogeneity part) when you require data reduction because of too small an effective sample size for the number of covariates to adjust for. Propensities balance because of the following reasoning. Suppose that accounting for a propensity to treat of 0.3 still resulted in an imbalanced age distribution between treatments. That implies that age was mismodeled when specifying the propensity model. If age were properly modeled, once adjusting for propensity age no longer predicts treatment choice. But then the outcome heterogeneity problem rears its ugly head.

• @StatsStudent, Frank , thanks as always, this is a great explanation. Please correct me if I am wrong. So at high level propensity score, is a method or framework that makes treatment independent of covariates. What will then remain is the effect of treatment on outcome and the effect of covariates on outcome. And this effect of covariates and treatment, acting separately on outcome will make the treatment effects collapsible. Is this correct ? Oct 30, 2021 at 19:26