# Is it possible of overfit using Propensity score matching with the MatchIt R package?

I have a very large patient cohort and I am trying to define cases and controls whilst minimizing selection bias. Further down the line, I am using Cox regression to assess the efficacy of particular drugs. This means that I will have calculated survival time to remission (to be mentioned later).

Depending on the drug in question I can get some awful selection bias. For example, topiramate is prescribed for patients with persistent recurrent headaches. A random sample of controls is unlikely to find cases where the burden of disease match that of those patients on topiramate.

I have therefore used propensity score matching to mitigate this concern. My matching covariate (what defines a case or control) is the presence or absence of a drug-of-interest. The accompanying covariates are twelve categorical variables each represents whether a patient had a particular comorbidity associated with a disease of interest (e.g., hypertension is associated with the disease of interest headache). I then have a number of continuous covariates: age, number of headaches in a given year, number of emergency neurology referrals. Combining these together I am able to generate my cases and controls using the R MatchIt package.

I am not a statistician by trade so this is where my understanding stops and my question begins:

1. Is it possible to over/under match using the MatchIt (propensity score matching) package? i.e., too many covariates or too few covariates.

2. What is the best way to understand what would be important to use as a MatchIt covariate? At the moment I am using a clinician's experience and published comorbidities associated with the disease to be treated. A round-table discussion with a group of clinicians brought about the suggestion of seeing whether there is any correlation between the survival time of a random sample from the cohort with the results from each of the covariates in the MatchIt procedure. e.g.,

survival_time ~ age survival_time ~ numEmergencyVisits

I don't think this is the right approach. And to be honest, selecting the matchIt covariates based on published links between comorbidities and the disease of interest should be enough.

I would appreciate the thoughts of those with a background in stats.

Regarding assessment of the propensity score model: the goal of matching is to create balance between the treated an untreated on the relevant covariates. You can do this either by trying various propensity score models and matching algorithms that yield balance or by attempting to specify a well-justified propensity score model that comes close to modeling the true propensity score. See some discussion about the distinction here. This matter is well-described in Ho, Imai, King, & Stuart (2007). The goal of the propensity score is to create balance, not achieve good fit. It's often better to model the propensity score in way that would normally be considered overfit if you were to try to interpret the propensity score model used than to ensure a parsimonious and theoretically justified model. There is a large literature on assessing balance, most of which is summarized in the documentation for the R package cobalt and my answer here.
There are so many ways of estimating propensity scores and using them that I really don't think someone with no expertise on this matter should be performing this analysis without the guidance of a trained statistician. Matching may not be (and likely isn't) the best method for you to use anyway. If you want to do as little thinking and decision-making as possible, I recommend you look into targeted maximum likelihood estimation of causal effects. The package survtmle implements this method. Whatever you do, do not choose a method just because it's what you know or it's what other researchers use, and do not attempt to perform a complex analysis without the help of a statistician.