1
$\begingroup$

I have two surveys of households in the same metropolitan areas about the number of transit trips they took. I would like to compare the change in number of transit trips taken by households, between the two periods. In addition to this pre-test/post-test, I would like to include a control group. I propose to match my control group to my treatment group using propensity score matching. And then to compare changes over time between my control group and my treatment group. HOWEVER the households in the two surveys are not the same, and so I need a mechanism to demonstrate equivalence between the two waves. It has been suggested that I use propensity score matching to do so. Is this valid? I am uncertain.

$\endgroup$
4
$\begingroup$

It sounds like you have a few different things going on here all which add some complexity. On the one hand, it sounds like you are performing a complex sample survey of a finite population. This in and of itself has its own challenges. Among them are things like finite population corrections, weighting, clustering and possible stratification, and design effect modifications. And on the other hand you have some some causal inference analyses that you are trying to carry out through the use of a pre-post analysis (some might called this a fixed effects analysis), and then you are intending to carry out a separate analysis using treatment and control groups, but you want to match on propensity scores.

This all sounds fine, but, again, it's somewhat complex, and you may need to seek a statistical consultant to help you out with the nuanced analyses.

The pre-post analysis will allow you to control for within-household variables that do not change over time, and the separate control/treatment analysis will allow you to understand/measure the change between treated and untreated groups while accounting for time-varying covariates. This essentially amounts to a "difference in difference" (DD) analysis if done appropriately. But as you've correctly thought out, one problem with the difference in difference analyses is that the treatment and control groups could differ in important ways that might affect their trends over time (one critical assumption of DD analysis is that the trends between the the two groups would remain parallel if no treatment were applied). To avoid this problem, you could indeed attempt to perform a propensity score matching on treamtents and controls. But again, this adds to a level of complexity and this procedure is really not that well investigated. See here, here, and here for a few exceptions.

The article by Stuart will probably be very helpful as the weights developed through that method could be used to modify survey weights and adapted for your needs.

One thing that you'll need to think through is which variables will you match on? Will you only be matching on variables you obtain once a survey is returned or on variables that are on the sample frame? If you intend to match on variables from the survey, how will you handle non-response and prevent it from further biasing results if non-respondents are systematically different from respondents?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.