Adjust for everything you have in propensity score?

Question

I have a methodological question, and therefore no sample dataset is attached.

I'm planning to do a propensity score adjusted Cox regression that aims to examine whether a certain drug will reduce the risk of an outcome. The study is observational, comprising 10,000 individuals.

The data set contains 60 variables. I judge that 25 of these might affect treatment allocation. I would never adjust for all 25 of these in a Cox regression, but I've heard that you can include that many variables as predictors in a propensity score and then only include the propensity score subclass and treatment variable in the Cox regression.

(covariates that will not be equal after prop score adjustment would of course have to be included in the Cox regression).

Bottom line, is it really smart to include that many predictors in the prop score?

---

@Dimitriy V. Masterov Thank you for sharing these important facts. On the contrary to books and articles considering other regression frameworks, I don't see any (reading Rosenbaums book) guidelines on model selection in propensity score analyses. While standard textbooks / review articles seem to always recommend stringent variable selection and keeping the number of predictors low, I haven't seen much of this discussion in prop score analyses. You write: (1) "Theoretical insight, institutional knowledge, and good research should guide selection of Xs". I agree but there are circumstances where we have a variable at hand and don't really know (but it might be possible) if the variable effects either treatment allocation or outcome. For example: should I include kidney function, as measure by filtration rate, in a prop score aiming to adjust for statin treatment. Statin treatment has nothing to do with kidney function and I have already included an array of variables that will effect statin treatment. But it is still tempting to include kidney function; it might adjust even more. Now some would say that it should be included because it effects outcome, but I could give you another example (such as the binary variable urban / rural living) of a variable that don't effect treatment nor outcome, as far as we know. But I would like to include it, as long as it don't effect the prop score precision. (2) "Including Xs affected by the treatment, either ex post or ex ante in anticipation of treatment, will invalidate the assumption". I'm not sure what you mean here. But if I study the effect of statins on cardiovascular outcome, I will include various measurements of blood lipids in the propensity score. Blood lipids are effected by the treatment. I guess I misunderstood this statement.

@statsRus thank you for sharing the facts, particularly what you call "a note on selecting inputs". I think I reasons much the same way you do.

Unfortunately prop score methods discuss various adjustment strategies instead of model selection strategies. Perhaps model fit is not important. If that is the case, I would adjust for every available variable that might effect outcome and treatment allocation the slightest. I am not a statician, but if model fit is of no importance then I would like to adjust for all variables that might affect treatment allocation and outcome. This would in many cases mean including variables that will be effected by the treatment.

Furthermore, some people suggest that the subsequent Cox regression should only include the treatment variable and prop score subclass. While others suggest that the cox adjustment should include the prop score additionally to all other variables that you would adjust for.

Answer 1

I've personally been asking this question for at least 5 years since for me it's the "big" practical question for using propensity score matching on observational data to estimate causal effects. This is a superb question and there's a subtle disagreement that runs deep in the statistics versus computer science communities.

From my experience statisticians tend to advocate "throwing the kitchen sink" of observable inputs into the estimation of the propensity score, while computer scientists tend to advocate a theoretical reason for the inputs (though statisticians may occasionally mention the importance of theory in justifying selection of inputs into the propensity score model). The difference, I believe, stems from the fact that computer scientists (in particular Judea Pearl) tend to think of causal in terms of directed acyclic graphs. When viewing causality through directed acyclic graphs, it's fairly easy to see that you can condition on a so-called "collider" variable, which may "un-block" backdoor paths and actually induce bias into your estimation of a causal effect.

My takeaway? If you have solid theory on what affects selection into the treatment, use that in the propensity score estimation. Then conduct a sensitivity analysis to determine how sensitive your estimate is to unobserved confounding variables. If you have almost no theory to guide you, then throw in the "kitchen sink" and then conduct a sensitivity analysis.

A note on selecting inputs for the propensity score model (this may be obvious but it's worth noting for others unfamiliar with estimating causal effects from observational data): Don't control for post-treatment variables. That is, you want your inputs in the propensity score model to be measured before the treatment and your outcome to be measured after the treatment. In observational data this practically means that you need three waves of data, with a detailed set of baseline of covariates, treatment measured in the second wave, and the outcome measured in the final wave.

Answer 2

In the absence of subject matter knowledge, overinclusion of variables is generally better than underinclusion, and there is little reason to do model selection to build a PS. What is more important is to build a flexible model. My default approach is to spline every continuous variable and to not look at $P$-values for variables in the PS, i.e., I use a flexible additive logistic regression model.

There are many advantages of covariate adjustment using the logit PS. I typically spline the logit of PS to include as a multiple degree of freedom adjustment variable, after doing due diligence regarding non-overlap regions. See http://www.citeulike.org/user/harrelfe/article/13340175 and http://www.citeulike.org/user/harrelfe/article/13265389 and more articles in http://www.citeulike.org/user/harrelfe/tag/propensity-score.

You have to be sure to also include as separate covariates the likely strong predictors of $Y$ as PS is just for bias adjustment, not for capturing outcome heterogeneity.

I am dubious of any matching method that results in discarding matchable observations or that is highly dependent on dataset order. Discarded observations have a lot to say about how covariate effects should be estimated.

Answer 3

Because the propensity score model is purely predictive - you're not interested in any coefficients - I've always understood it than you can hurl in all your variables that affect both cohort entry and outcome. You can twist these variables as you wish - square them, root them, all types of interactions, etc. etc. - as long as you're increasing the predictive quality of your model.

In theory, you shouldn't even have to worry about hold-out data for your predictive model as you have no desire to generalise these results past your sample (basically, the risk of 'overfitting' isn't a problem). Finally, you don't have to limit yourself to logistic regression; as you're modelling a binary output, you might even use a GAM model - basically, anything to improve the prediction rates.

( I must add as a contrary note to @statsRus' point on use: in my experience it's the computer scientists who use all variables while the statisticians who carefully consider each one. I guess different work backgrounds produce different working habits. )

As for use of the score, it's generally discouraged to use it as a covariate - it has less impact - and certainly not alongside the variables used to make the scoring variable. An argument might be made if, in the propensity score, you categorised a continuous variable - age for instance - where you might then include the continuous version in the model but really, don't categorise the variable the first place...

Using the score for matching (with calipers - especially variable 1:N matching) is popular but I believe the most impactful technique is as Inverse Proportional Treatment Weights (IPTW) - although I've not used this method and I can't remember how it works.

Try looking at Peter C. Austin's work at the University of Toronto - he's written loads of papers on propensity scores. Here's one on matching for instance.

Answer 4

Theoretical insight, institutional knowledge, and good research in the field should be your guide about what $X$s to match on. There is no deterministic variable selection procedure that will tell you which variables to choose.

Here are some general guidelines. The Conditional Independence Assumption (CIA) will be satisfied if $X$ includes all of the variables that affect both (not either, but both) participation and outcomes. Including $X$s affected by the treatment, either ex post or ex ante in anticipation of treatment, will invalidate the assumption. For example, if an agent knows that the vaccine is coming, he may adjust his pre-shot behavior. Including instruments – variables that affect participation and not outcomes – is also a bad idea. They will not help with selection bias and may worsen the support problem drastically. For example, if some people are encouraged to take up treatment, you don't want to condition on that. The inclusion of irrelevant variables in the propensity score specification can increase the variance since either some treated have to be discarded from the analysis or control units have to be used more than once or because the bandwidth has to increase. In short, the kitchen sink approach is definitely not recommended.

The CIA cannot be tested without experimental data or "over-identifying" assumptions (as in the case of the pre-program test or other false placebo tests). If you have enough historical data, I would definitely try the latter on your carefully curated set.

---

Response to edit:
I can't comment on the kidneys since that is too far outside my area (other than pies, which I know something about). Urban seems like a variable that affects both participation and outcome through the costs associated with travel to the hospital for treatment and examination. It might pick up some of the unobservables that keep us up at night. The anticipation story I have in mind is that people may adjust their behavior if they know they will be treated in the future, for example by changing their diets.