After matching on propensity score, e.g 1:1 matching, you obtain a matched subset of your data.

The built-in functions in the Matching package, as a prominent example, compares groups before matching by use of non-paired t-test but then switches to the paired t-test to compare groups after matching.

Publications, in medical journals at least, obviously fail to report what type of t-test they have chosen. As far as I can tell, I haven't seen a single one using the paired test, which contrasts rather strongly against J Sekhons Matching package.

I "did like the rest" and used a non-paired t-test to compare continuous variables in a propensity score matched cohort (1:1 matched).

Was this wrong?

  • 1
    $\begingroup$ I think they don't say because they think it's obvious that cases are now matched so you use a paired $t$. I suppose the independent-sample test is a conservative approach (so not exactly wrong), where you only use the propensity scores to ensure that you have comparable populations. I'll be interested to see what the propensity-score experts say. $\endgroup$ – Russ Lenth Dec 23 '14 at 23:47
  • $\begingroup$ yes, it might be so obvious it isn't spelled out. But I'm afraid that many published baseline tables (which are extremely important in this scenario) are simply based on independent samples t test; particularly since these tests are easily performed with the tableone package. Where's our experts on Christmas? :-) $\endgroup$ – Adam Robinsson Dec 24 '14 at 16:26
  • $\begingroup$ I did find a related post - stats.stackexchange.com/questions/25392/… - that includes a quote from a reference that recommends an independent-samples test. "No reason to believe they are correlated just because they share a propensity score." $\endgroup$ – Russ Lenth Dec 26 '14 at 2:33

I personally find results are very similar when you use paired and unpaired tests. Yet, my recommendation, built upon studying quite extensively the topic, and following authoritative sources, such as this one from Austin, is now to use tests that recognize the clustering features of the dataset.

Thus, if I am using propensity score quantiles (eg quintiles), or propensity matched pairs, I routinely use meglm or xtgee in Stata, for continous or categorical variables, and stratified Cox proportional hazard analysis for survival analysis.

Specifically, the following excerpt, also from Austin, is very clear:

When estimating the statistical significance of treatment effects, the use of methods that account for the matched nature of the sample is recommended (Austin, 2009d, in press-b). Accordingly, McNemar's test was used to assess the statistical significance of the risk difference. Confidence intervals were constructed using a method proposed by Agresti and Min (2004) that accounts for the matched nature of the sample. The number needed to treat (NNT) is the reciprocal of the absolute risk reduction. The relative risk was estimated as the ratio of the probability of 3-year mortality in treated participants compared with that of untreated participants in the matched sample. Methods described by Agresti and Min were used to estimate 95% confidence intervals.

We then estimated the effect of provision of smoking cessation counseling on the time to death. Kaplan-Meier survival curves were estimated separately for treated and untreated participants in the propensity score matched sample. The log-rank test is not appropriate for comparing the Kaplan-Meier survival curves between treatment groups because the test assumes two independent samples (Harrington, 2005; Klein & Moeschberger, 1997). However, the stratified logrank test is appropriate for matched pairs data (Klein & Moeschberger, 1997).

Finally, we used a Cox proportional hazards model to regress survival time on an indicator variable denoting treatment status (smoking cessation counseling vs. no counseling). As the propensity score matched sample does not consist of independent observations, we used a marginal survival model with robust standard errors (Lin & Wei, 1989). An alternative to the use of a marginal model with robust variance estimation would be to fit a Cox proportional hazards model that stratified on the matched pairs (Cummings, McKnight, & Greenland, 2003). This approach accounts for the within-pair homogeneity by allowing the baseline hazard function to vary across matched sets.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.