Is it possible of overfit using Propensity score matching with the MatchIt R package?

Question

I have a very large patient cohort and I am trying to define cases and controls whilst minimizing selection bias. Further down the line, I am using Cox regression to assess the efficacy of particular drugs. This means that I will have calculated survival time to remission (to be mentioned later).

Depending on the drug in question I can get some awful selection bias. For example, topiramate is prescribed for patients with persistent recurrent headaches. A random sample of controls is unlikely to find cases where the burden of disease match that of those patients on topiramate.

I have therefore used propensity score matching to mitigate this concern. My matching covariate (what defines a case or control) is the presence or absence of a drug-of-interest. The accompanying covariates are twelve categorical variables each represents whether a patient had a particular comorbidity associated with a disease of interest (e.g., hypertension is associated with the disease of interest headache). I then have a number of continuous covariates: age, number of headaches in a given year, number of emergency neurology referrals. Combining these together I am able to generate my cases and controls using the R MatchIt package.

I am not a statistician by trade so this is where my understanding stops and my question begins:

1. Is it possible to over/under match using the MatchIt (propensity score matching) package? i.e., too many covariates or too few covariates.

2. What is the best way to understand what would be important to use as a MatchIt covariate? At the moment I am using a clinician's experience and published comorbidities associated with the disease to be treated. A round-table discussion with a group of clinicians brought about the suggestion of seeing whether there is any correlation between the survival time of a random sample from the cohort with the results from each of the covariates in the MatchIt procedure. e.g.,

   survival_time ~ age survival_time ~ numEmergencyVisits

I don't think this is the right approach. And to be honest, selecting the matchIt covariates based on published links between comorbidities and the disease of interest should be enough.

I would appreciate the thoughts of those with a background in stats.

Answer 1

First, I would caution anyone without a background in applied statistics from performing advanced analyses like propensity score matching. The ease of the software makes it seem like the procedure itself is easy, when in fact there are many considerations required to make a valid inference. I'm also pretty skeptical of guides for practice published in non-statistical journals by non-statisticians, as these tend to focus on the use of software rather than consideration of the nuances of the analysis. I'm sure you could find a biostatistician who would be willing to help you implement best practices.

That said, you bring up two major issues in propensity score analysis: covariate selection and assessment of the propensity score model. These are both huge topics about which there is ongoing research. I'll give you some pointers and some literature that can help you make your decisions.

Regarding covariate selection: if you want an unbiased estimated of the treatment effect, you need to eliminate confounding without introducing bias. To eliminate confounding, you need to control for a sufficient set of variables that blocks all the backdoor paths from treatment to the outcome. A backdoor path is a causal pathway that involves a common cause of treatment assignment and variation in the outcome. Substantive research can be illuminating, but it's better to be as conservative as possible by including as many relevant variables as possible without inducing bias. The types of variables you should include are those that affect the outcome and are not affected by the treatment. You should also include variables that are known to affect both selection into treatment and variation in the outcome. Do not include variables that could possibly be affected by the treatment or that affect selection into treatment but are otherwise unrelated to the outcome. See Brookhart et al. (2006) for specific discussion of variable selection for propensity score models and Elwert (2010) for discussion of a more complete theory for adjusting for confounding. Critically, failing to include relevant confounding variable can bias your effect, and medical research may not have found all of them, so I would warn you against excluding covariates just because there isn't a medical paper documenting the confounding relationship. Prior research can be used as positive evidence to justify the necessity of a variable's inclusion, but there is likely no negative evidence that can justify a variable's exclusion except for a demonstration of its lack of effect on the outcome.

Regarding assessment of the propensity score model: the goal of matching is to create balance between the treated an untreated on the relevant covariates. You can do this either by trying various propensity score models and matching algorithms that yield balance or by attempting to specify a well-justified propensity score model that comes close to modeling the true propensity score. See some discussion about the distinction here. This matter is well-described in Ho, Imai, King, & Stuart (2007). The goal of the propensity score is to create balance, not achieve good fit. It's often better to model the propensity score in way that would normally be considered overfit if you were to try to interpret the propensity score model used than to ensure a parsimonious and theoretically justified model. There is a large literature on assessing balance, most of which is summarized in the documentation for the R package cobalt and my answer here.

There are so many ways of estimating propensity scores and using them that I really don't think someone with no expertise on this matter should be performing this analysis without the guidance of a trained statistician. Matching may not be (and likely isn't) the best method for you to use anyway. If you want to do as little thinking and decision-making as possible, I recommend you look into targeted maximum likelihood estimation of causal effects. The package survtmle implements this method. Whatever you do, do not choose a method just because it's what you know or it's what other researchers use, and do not attempt to perform a complex analysis without the help of a statistician.