1
$\begingroup$

I'm trying to use the Match() function from the Matching package in R to do a propensity score analysis.

My outcome of interest is a binary variable (0/1). My treatment is also a binary variable (0/1). In addition, I have a number of other variables that I want to control for in this analysis.

First, I fit a logistic regression to define a propensity score for the treatment:

glm1 = glm(Treatment ~ variable1 + variable2 + variable3 + ..., 
           data=dataset, family="binomial")

Then, I used the Match function to estimate the average treatment effect on the treated:

rr1 = Match(Y = Outcome, Tr = Treatment, X = glm1$fitted)

Finally, I called for a summary:

summary(rr1)

My question is how to interpret the output. I get:

Estimate... -0.349,
AI SE... 0.124,
T-stat... -2.827,
p.val... 0.005

What does this mean? In particular, what is Estimate? The documentation says it's "The estimated average causal effect." But what are the units? Can I interpret this to mean that the treatment reduced the outcome by a relative 35%? Or by an absolute 0.35? Or do I need to exponentiate?

Any help on the interpretation would be much appreciated!

$\endgroup$

migrated from stackoverflow.com Aug 18 '14 at 1:26

This question came from our site for professional and enthusiast programmers.

  • $\begingroup$ I think it was a mistake to migrate this question to CV from SO, since it doesn't sound like you're asking about the statistics, just the implementation. $\endgroup$ – shadowtalker Aug 18 '14 at 2:11
  • 2
    $\begingroup$ @ssdecontrol I disagree, although there is usually a gray area between the two sites with R questions. There were no actual coding problems to solved here. No errors, no unexpected or incorrect results. The OP simply didn't understand the statistical meaning of the output. $\endgroup$ – joran Aug 18 '14 at 14:59
  • $\begingroup$ Any luck at understanding what the estimate, the t-stat or the p value means? I am confused by the documentation as well $\endgroup$ – stats_nerd Jul 8 at 12:18
1
$\begingroup$

So the output is

Estimate... -0.349,
AI SE... 0.124,
T-stat... -2.827,
p.val... 0.005

You did the matching presumably because you'd like to interpret the difference in outcome for treatment and control as a causal effect, i.e. as the change in the dependent variable caused by treatment, and you don't necessarily trust a big regression with controls to work out for you (though you do trust that you've got all the causes of treatment assignment bundled into the propensity score model).

In your case I guess that the dependent variable is a probability. If so then the matching analysis says that that probability is 0.35 less due to treatment - so an absolute 0.35 because you're computing a difference. This difference is computed after your data set is matched, pruned, etc. as well as it can to balance covariates over treatment and control cases. Actually you'd want to check that balance using other functions in the package before just trusting the summary output.

You have a lot of control over what 'good matching' means, though you've gone with the defaults which are, I believe to calculate an average treatment effect (ATE), not use calipers, etc. You can see the defaults on the relevant help page. So that's the Estimate here.

The AI SE is a matching corrected standard error due to Abadie and Imbens (hence the name AI). The t-stat and p.value are interpretable as usual, though corrected with that standard error. The details of AI standard errors you can find in A and I's original paper.

$\endgroup$
  • $\begingroup$ @forecaster I don't think that a guess from "some guy on the Internet" that the phrase "causal effect" is a probability is very firm evidence. It's your responsibility to look at the documentation and the code to see what the function Match (which in turn calls Rmatch to calculate the "estimate") is actually doing. When I look at the code it doesn't appear to be a probability of any sort, but rather a weighted (linear) regression coefficient. This is still a guess, so now you have divergent guesses from two "guys on the Internet". $\endgroup$ – DWin Jul 18 '15 at 18:16
  • 1
    $\begingroup$ @DWin It seems you're having a bad day. Hope it improves. $\endgroup$ – conjugateprior Jul 18 '15 at 18:25
  • $\begingroup$ And I hope @forecaster investigates the context in which the author of the Match package uses the phrase "causal effect". sekhon.berkeley.edu/causalinf/fa2014/Slides/Slides1_OLS/… $\endgroup$ – DWin Jul 18 '15 at 19:04
  • $\begingroup$ @DWin, I don't know why you keep referring me when conjugateprior provided the answer. $\endgroup$ – forecaster Jul 18 '15 at 21:32
  • 1
    $\begingroup$ On a) it's true that I guessed that the DV was a probability. The ATE computed is, on the other hand, not a probability but some kind of adjusted difference of DV means. When the DV is a probability then it is a probability difference. When not, it is a difference in something else. It doesn't actually matter whether the DV is a probability or not for understanding the output. It's just a kind of difference. And yes, the difference is realised in a regression-like manner. Not so surprising, since regression coefficients for dummy variable Treatment variables are, at root, weighted differences. $\endgroup$ – conjugateprior Jul 19 '15 at 7:52
0
$\begingroup$

R functions typically return answers on the same scale as the inputs, so I would guess it's difference in logits. Moreover, functions like this typically don't/can't check where their inputs came from (i.e. whether it's a glm or an lm or something else) so they operate agnostically, which in this case would be to just take a difference in mean $Y$.

When in doubt in situations like this you can always check the source code. Type the name of the function without () after it. Or in Rstudio, highlight the name of the function and ctrl+click.

$\endgroup$
  • $\begingroup$ This is reasonable general advice, but this is one of those cases where one has to know something about what the package is doing. (And you're correct about it being a difference of mean Y in this case, albeit with some pretty significant preprocessing beforehand). $\endgroup$ – conjugateprior Jul 18 '15 at 15:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.