2
$\begingroup$

In the MatchIt tutorial Getting Started a multiple linear regression is used to estimate the effect of the treatment on income. But after I have a matched sample, can I just do a t-test instead of a multiple linear regression to estimate an effect (in the form of Cohen's d)? Compare all people with treatment "yes" in the matched sample with all people with treatment "no" in the sample. Then the result would be something like "People in the treatment group have a higher income in the magnitude of d=.40"

The reason I ask is, I try to replicate a paper which used MatchIt. It did not report the findings like I expected after reading "Getting Started".

Recommended reporting:

To estimate the treatment effect and its standard error, we fit a linear regression model with 1978 earnings as the outcome and the treatment and the covariates as additive predictors and included the full matching weights in the estimation. The coefficient on the treatment was taken to be the estimate of the treatment effect. The lm() function was used to estimate the effect, and a cluster-robust variance as implemented in the vcovCL() function in the sandwich package was used to estimate its standard error with matching stratum membership as the clustering variable.

The estimated effect was $1980 (SE = 756.1, p = .009), indicating that the average effect of the treatment for those who received it is to increase earnings.

How the results were reported:

To provide a more stringent test of the effect of military training on personality, we next ran a propensity-score analysis, in which we controlled for a large number of potentially confounding covariates (Table 1). As in the previous analysis, results showed that military recruits had lower levels of agreeableness than individuals who chose civilian community service (d = −0.15, p < .05).

So I suspect they used a different statistical test. Or they somehow turned regression coefficients into Cohen's d. But as far as I know, those are not really the same.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

A regression of a continuous outcome on a single binary predictor is identical to a two-sample t-test. So it is straightforward to turn a regression coefficient into Cohen's d in this scenario. The reason the MatchIt documentation recommends regression is that with regression, it's more straightforward to include weights (which are necessary for most forms of matching other than 1:1 pair matching without replacement), correct standard errors (i.e., cluster-robust), and additional covariates. There is no good reason to use a t-test: the syntax of t.test() is the same as that of lm() but it is far less flexible.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you, I have a follow up question. I want to replicate the latter finding (d=-.15), and I do a power analysis first (to determine the required sample size). In G*Power, you can base the power analysis on several statistical test. Can I base it on “difference of two independent means” instead of “linear multiple regression”? Because the authors did not provide the partial R^2 of the treatment predictor, which I would need for a calculation based on "linear multiple regression". $\endgroup$
    – E_H
    May 14, 2021 at 8:04
  • 1
    $\begingroup$ As I said, the test for the difference between two independent means is identical to the test for linear SINGLE regression. So you can just use the coefficient on treatment as the difference in treatment group means. $\endgroup$
    – Noah
    May 15, 2021 at 5:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.