Propensity scores in logistic regression models

Question

I have a query after reading a paper, which is about the effectiveness of a medical device. In summary, what the authors did was 1. Generating a propensity score using a multivariable logistic regression based on x number of covariates to see the probability of receiving the medical device. 2. They subsequently evaluated the association between the device and a pre-specified patient outcome Y using logistic regression models. Their model was as follows:

Y ~ propensity score + presence of device + variable Z

in which variable Z is an ordinal variable

3. They then assessed for effect measure modification of the medical device on patient outcome depending on the level of variable Z by introducing an interaction term significance of the interaction in the model

My queries are as follows:

1. What is the advantage of not introducing variable Z when generating the propensity scores? Why did they have to use the propensity score as one variable and variable Z separately? Since they were interested the effect modification, it would signify that variable Z did play a role in influencing the presence of the medical device and the outcome, therefore putting it into the propensity score analysis should be valid?

2. Are there any limitations in introducing the propensity score into the logistic regression model, as opposed to using the propensity score to do matching (e.g. greedy/ optimal, with/without replacement)? Then finding out the relationship using odds ratios?

Answer 1

I want to add on to @StatsStudent's answer.

There are a few ways to control for a confounder. One is by including that confounder in a well-formed propensity score and conditioning on the propensity score. Another is conditioning directly on the confounder. If $Z$ is a confounder, then generally it belongs in the propensity score model. However, it might be possible to get a well-formed propensity score model without $Z$, in which case it would be sufficient to control for $Z$ by stratifying on it (i.e., including it in the outcome regression model, possibly interacting it with treatment). Indeed, Eeren & de Rooij (2015) found through simulations that even if treatment assignment depended on an effect modifier $Z$, it was preferable to exclude $Z$ from the propensity score model but include it in the outcome model as a treatment effect modifier, just as the authors of the paper you're referencing did.

In general, this is a risky move, and it's theoretically confusing why Eeren & de Rooij found what they did. It may simply be an artifact of their simulation. That said, if one has confidence that $Z$ is not a confounder but is an effect modifier, then there is no requirement to include it in the propensity score model. It goes in the outcome model to estimate the effect modification relationship, but it isn't necessary to additionally control for it through the propensity score.

A major limitation of using the propensity score in a logistic regression model instead of performing matching or weighting is that the estimand is not the average treatment effect in the population. Odds ratios are not collapsible, which means the conditional odds ratio (i.e., when other variables are in a regression model) differs from a marginal odds ratio (when no other variables are in the regression model). Including the propensity score in a logistic regression model for the outcome means the interpretation of the treatment effect is the odds ratio conditional on the propensity score (i.e., for a population held at a given propensity score, what would be the ratio of the odds of the outcome event were the population to be assigned treatment vs. assigned control). Instead using matching or weighting to adjust for confounding and only including the treatment in the outcome model would yield a marginal odds ratio (i.e., for the entire population, what would be the ratio of the odds of the outcome event were the population to be assigned treatment vs. assigned control). The latter is typically what researchers want, which is what motivates using these methods over including the propensity score in the outcome model.

So, the authors took two risks here: first, by not including the effect modifier $Z$ in the propensity score model, they assumed $Z$ was not a confounder or that the results of Eeren & de Rooij justified its exclusion even if it was a confounder. Second, by using covariate adjustment for the propensity score instead of matching or weighting on the propensity score, they assumed the conditional odds ratio (what they got from the regression) was the same as the marginal odds ratio (what they probably wanted to estimate). Many other assumptions were made as well that could severely invalidate the results: they assumed no effect modification by the propensity score, they assumed the correct functional form for the relationship between the propensity score and the outcome, they assumed they had a well-formed propensity score, the list goes on... (and some of these were mentioned by @StatsStudent). In general I would be very wary of the results of this analysis.

---

Eeren, H. V., & de Rooij, M. (2015). Estimating Subgroup Effects Using the Propensity Score Method. Medical Care, 53(4), 8.

Answer 2

Using a propensity score in the manner that the authors used it is an incorrect application of the propensity score (this is a very common mistake that researchers do when applying propensity scores and you'll find it all over the medical literature, unfortunately). Instead, the generated propensity scores should have been used as either weights or used to create strata consisting of patients with similar propensity scores within which separate analyses could be carried out (or other similar matching strategies).

It's hard to say without knowing more about what variable $Z$ is, but there are a few different reasons why one might not include $Z$ in the propensity score model. When generating propensity scores, one is trying to determine a probability of being assigned to the treatment condition, given covariates that might be correlated with the decision to assign treatment (confounders). It's possible that variable $Z$ was not a variable that could have affected the treatment assignment (or possibly it was affected by the treatment and not vice versa), and not a covariate that is correlated with treatment assignment, so it would not make sense to include it in a logistic regression model to determine the probability of treatment assignment.

There are several limitations of using logistic regression models to determine propensity scores (as opposed to other methods of propensity score generation). Two of them are:

1. It may not be possible to capture complex relationships between the covariates and how they relate to the propensity of being assigned to the treatment group. For example, there may be complex interactions (e.g. $X_1^4 \times X_5^2$) that are not captured during the modeling process with a large number of covariates. In other words, it may be difficult and time consuming (another possible limitation) to determine the appropriate functional form of the propensity score model. Propensity score generation via logistic regression model can also be quite time consuming compared to automted machine learning methods such as Generalized Boosted Methods (GBM) (Ridgeway, 1999). This itself could be considered a drawback.
2. Generating propensity scores from logistic regression often calls for including any covariate that could potentially influence the treatment assignment. This often includes a very large number of correlated covariates that, when combined with few subjects, could make the logistic regression results, typically generated via reweighted least squares, numerically unstable (Ridgeway, McCaffrey, Morral, et al., 2017).

That all being said, it is generally less important what method you use to generate propensity scores than it is determining which covariates are to be used in the propensity score generation (Steiner and Cook, 2013). In addition, what one should really be paying attention to are the covariance balance score-cards. The authors should have provided covariate balance diagnostics. If the authors were able to achieve good balance, then this is quite a good indication that the the propensity scores were well-modeled and achieved the desired result (essentially making the treatment groups comparable).

Finally, if you are interested in a good review of pros and cons of different of the various techniques for the application of propensity scores (e.g. pros and cons of greedy-matching vs. optimal for example) see the article "Methodological Considerations in Implementing Propensity Score Matching" by Haiyan Bai that appears in Chapter 4 of Propensity Score Analysis: Fundamentals and Developments edited by Wei Pan and Haiyan Bai. In addition, the article by Garrido, Kelley, Paris, et al. provides a good overview of the pros and cons of different techniques that make use of propensity scores.