4
$\begingroup$

I have a tough challenge using the DID. I have only 2 year data set, 2010 and 2015. The first is the baseline and the latter is follow-up year. In order to obtain true causal effect using DID, I should ensure that common trend assumption is satisfied, which is in my case It is impossible to test due to data limitation. Another case is that the covariates between treated units and the controlled units are very likely unbalance. Basically I compare apples with oranges.

This condition makes me to use what I called as propensity score-weighted DID. So, I run a probit regression first to obtain propensity scores for each units using baseline data. I use the propensity score as weight to each sample in implementing the DID which is a panel data set-based. The weight for treated units is 1 and for the controlled units is p/(1-p) where p is propensity scores of each controlled units. Then I will get DID estimator.

Did I do the right procedures? Thank you.

$\endgroup$
4
$\begingroup$

At first glance, both the propensity score matching DID (PSM DID) and the inverse probability weighting (IPW DID) that you want to do are sensible ways to approach this: matching and weighting will take care of the selection into treatment based on observables, and the DID will deal with selection on unobservables as long as the bias from it is time-invariant, conditional on covariates. Matching is just an attempt to approximate what weighting is doing directly. If you're just worried about covariate imbalance, you can include them as controls, as long as they are variables that affect both treatment and the outcome, and are not affected by the treatment itself or by anticipation of treatment.

Also, as an aside, you can never test the parallel paths or trends/growth assumptions, even if you had more data. All you can say is that is seems to hold in the pre-treatment period, so you would be more willing to believe it holds in the post-treatment future. But is is still an assumption.

In the example below, your IPW approach will produce estimates that match what Stata's own teffects ipw would give you for the ATET/ATT and ATE. I will also demonstrate some alternative matching methods that you might find useful as a robustness check. Why? With IPW, things can go south if the estimated propensity scores are near zero or 1 since you would be dividing by small numbers. This is not the case in this example, and so the estimates are fairly similar. Asymptotically, both PSM and IPW should give you the same answer (though I cannot recall where I saw this result).

Below I will employ a user-written command that can do two-period Kernel PSM DID called diff. I will also do a regression version with the kernel-based weights from the command. Finally, you can also trick matching estimators, like user-written psmatch2 or Stata's own teffects ipw, to do DID with two periods, or you can also do a regression version of ipw.

Here's the annotated example:

. set more off

. estimates clear

. use http://fmwww.bc.edu/repec/bocode/c/CardKrueger1994.dta, clear
(Dataset from Card&Krueger (1994))

. /* Need to balance the panels so that all the methods use the same sample */
. drop if missing(fte)
(19 observations deleted)

. bys id (t): keep if _N==2
(23 observations deleted)

. 
. /* (1) -diff- to get ATT */
. diff fte, treated(treated) period(t) id(id) cov(bk kfc roys) kernel ktype(gaussian) support bw(0.05) cluster(id) robust // report
KERNEL PROPENSITY SCORE MATCHING DIFFERENCE-IN-DIFFERENCES
    Estimation on common support
    Matching iterations...
..............................................................................................................................................................................
> ............................................................................................................................................
DIFFERENCE-IN-DIFFERENCES ESTIMATION RESULTS
Number of observations in the DIFF-IN-DIFF: 778
            Baseline       Follow-up
   Control: 75             75          150
   Treated: 314            314         628
            389            389
------------------------------------------------------
 Outcome var.   | fte     | S. Err. |   t   |  P>|t|
----------------+---------+---------+-------+---------
Baseline        |         |         |       | 
   Control      | 20.471  |         |       | 
   Treated      | 17.069  |         |       | 
   Diff (T-C)   | -3.402  | 1.481   | -2.30 | 0.022**
Follow-up       |         |         |       | 
   Control      | 17.792  |         |       | 
   Treated      | 17.518  |         |       | 
   Diff (T-C)   | -0.274  | 1.067   | -0.26 | 0.798
                |         |         |       | 
Diff-in-Diff    | 3.128   | 1.388   | 2.25  | 0.025**
------------------------------------------------------
R-square:    0.02
* Means and Standard Errors are estimated by linear regression
**Robust Std. Errors
**Clustered Std. Errors
**Inference: *** p<0.01; ** p<0.05; * p<0.1

. assert _support==1

. 
. /* (2a) Regression Using Kernel Weights for ATE */
. reg fte i.treated##i.t [aw=_weights], cluster(id) robust
(sum of wgt is   1.2560e+03)

Linear regression                               Number of obs     =        778
                                                F(3, 388)         =       2.04
                                                Prob > F          =     0.1075
                                                R-squared         =     0.0193
                                                Root MSE          =        9.5

                                   (Std. Err. adjusted for 389 clusters in id)
------------------------------------------------------------------------------
             |               Robust
         fte |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
     treated |
         NJ  |  -3.401828   1.481415    -2.30   0.022    -6.314433   -.4892233
         1.t |  -2.679224   1.307741    -2.05   0.041    -5.250369   -.1080794
             |
   treated#t |
       NJ#1  |   3.128269   1.388281     2.25   0.025     .3987744    5.857763
             |
       _cons |    20.4711    1.39505    14.67   0.000     17.72829     23.2139
------------------------------------------------------------------------------

. xtset id t
       panel variable:  id (strongly balanced)
        time variable:  t, 0 to 1
                delta:  1 unit

. xtreg fte i.treated##i.t [aw=_weights], fe cluster(id) robust
note: 1.treated omitted because of collinearity

Fixed-effects (within) regression               Number of obs     =        778
Group variable: id                              Number of groups  =        389

R-sq:                                           Obs per group:
     within  = 0.0376                                         min =          2
     between = 0.0065                                         avg =        2.0
     overall = 0.0000                                         max =          2

                                                F(2,388)          =       2.57
corr(u_i, Xb)  = -0.1104                        Prob > F          =     0.0781

                                   (Std. Err. adjusted for 389 clusters in id)
------------------------------------------------------------------------------
             |               Robust
         fte |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
     treated |
         NJ  |          0  (omitted)
         1.t |  -2.679224   1.306897    -2.05   0.041     -5.24871   -.1097388
             |
   treated#t |
       NJ#1  |   3.128269   1.387385     2.25   0.025     .4005359    5.856002
             |
       _cons |   18.77018   .3468462    54.12   0.000     18.08825    19.45211
-------------+----------------------------------------------------------------
     sigma_u |  8.0179596
     sigma_e |  6.8873514
         rho |  .57541878   (fraction of variance due to u_i)
------------------------------------------------------------------------------

. 
. /* Create ATT and ATE Weights for regressions*/
. /* You can install -xfill- by typing net from https://www.sealedenvelope.com/ and clicking on the name */
. xfill _ps, i(id)

. gen double w_att = cond(treated==1,1,_ps/(1-_ps))

. gen double w_ate = cond(treated==1,1/_ps,1/(1-_ps))

. 
. /* (2b) Regression Using ATT and ATE IPW Weights */
. reg fte i.treated##i.t [pw=w_att] /*if _support==1*/, cluster(id) robust
(sum of wgt is   1.2560e+03)

Linear regression                               Number of obs     =        778
                                                F(3, 388)         =       1.48
                                                Prob > F          =     0.2187
                                                R-squared         =     0.0079
                                                Root MSE          =     9.4359

                                   (Std. Err. adjusted for 389 clusters in id)
------------------------------------------------------------------------------
             |               Robust
         fte |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
     treated |
         NJ  |  -1.897508   1.458586    -1.30   0.194    -4.765229    .9702128
         1.t |  -2.182001   1.170245    -1.86   0.063    -4.482815     .118813
             |
   treated#t |
       NJ#1  |   2.631046   1.259607     2.09   0.037     .1545361    5.107555
             |
       _cons |   18.96678   1.370783    13.84   0.000     16.27168    21.66187
------------------------------------------------------------------------------

. reg fte i.treated##i.t [pw=w_ate] /*if _support==1*/, cluster(id) robust
(sum of wgt is   1.5560e+03)

Linear regression                               Number of obs     =        778
                                                F(3, 388)         =       1.53
                                                Prob > F          =     0.2068
                                                R-squared         =     0.0086
                                                Root MSE          =     9.4409

                                   (Std. Err. adjusted for 389 clusters in id)
------------------------------------------------------------------------------
             |               Robust
         fte |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
     treated |
         NJ  |  -2.014025   1.455524    -1.38   0.167    -4.875727    .8476772
         1.t |  -2.247811   1.184075    -1.90   0.058    -4.575816    .0801949
             |
   treated#t |
       NJ#1  |   2.712311   1.275497     2.13   0.034     .2045602    5.220061
             |
       _cons |    19.1994   1.367706    14.04   0.000     16.51036    21.88845
------------------------------------------------------------------------------

. xtreg fte i.treated##i.t [pw=w_att], fe cluster(id) robust
note: 1.treated omitted because of collinearity

Fixed-effects (within) regression               Number of obs     =        778
Group variable: id                              Number of groups  =        389

R-sq:                                           Obs per group:
     within  = 0.0275                                         min =          2
     between = 0.0065                                         avg =        2.0
     overall = 0.0000                                         max =          2

                                                F(2,388)          =       2.21
corr(u_i, Xb)  = -0.1023                        Prob > F          =     0.1116

                                   (Std. Err. adjusted for 389 clusters in id)
------------------------------------------------------------------------------
             |               Robust
         fte |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
     treated |
         NJ  |          0  (omitted)
         1.t |  -2.182001   1.169489    -1.87   0.063    -4.481331    .1173282
             |
   treated#t |
       NJ#1  |   2.631046   1.258794     2.09   0.037     .1561344    5.105957
             |
       _cons |   18.01802   .3146986    57.25   0.000     17.39929    18.63675
-------------+----------------------------------------------------------------
     sigma_u |  8.0031528
     sigma_e |  6.6447787
         rho |  .59194417   (fraction of variance due to u_i)
------------------------------------------------------------------------------

. xtreg fte i.treated##i.t [pw=w_ate], fe cluster(id) robust
note: 1.treated omitted because of collinearity

Fixed-effects (within) regression               Number of obs     =        778
Group variable: id                              Number of groups  =        389

R-sq:                                           Obs per group:
     within  = 0.0287                                         min =          2
     between = 0.0065                                         avg =        2.0
     overall = 0.0000                                         max =          2

                                                F(2,388)          =       2.28
corr(u_i, Xb)  = -0.1037                        Prob > F          =     0.1032

                                   (Std. Err. adjusted for 389 clusters in id)
------------------------------------------------------------------------------
             |               Robust
         fte |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
     treated |
         NJ  |          0  (omitted)
         1.t |  -2.247811   1.183311    -1.90   0.058    -4.574314    .0786925
             |
   treated#t |
       NJ#1  |   2.712311   1.274674     2.13   0.034     .2061786    5.218443
             |
       _cons |   18.19239   .3186684    57.09   0.000     17.56586    18.81892
-------------+----------------------------------------------------------------
     sigma_u |  8.0054923
     sigma_e |  6.6977382
         rho |  .58824522   (fraction of variance due to u_i)
------------------------------------------------------------------------------

. 
. /* (3) psmatch2 for ATE and ATT */
. keep id fte treated t bk kfc roys

. qui reshape wide fte, i(id treated bk kfc roys) j(t)

. gen did_fte = fte1 - fte0

. psmatch2 treated bk kfc roys, outcome(fte0 fte1 did_fte) kernel kerneltype(normal) bw(0.05) common ate

Probit regression                               Number of obs     =        389
                                                LR chi2(3)        =       2.97
                                                Prob > chi2       =     0.3956
Log likelihood = -189.22405                     Pseudo R2         =     0.0078

------------------------------------------------------------------------------
     treated |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
          bk |   .1368372   .2190829     0.62   0.532    -.2925573    .5662318
         kfc |   .4092482   .2579891     1.59   0.113    -.0964012    .9148976
        roys |   .2448943   .2415267     1.01   0.311    -.2284893    .7182778
       _cons |   .6744898   .1889631     3.57   0.000     .3041288    1.044851
------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
        Variable     Sample |    Treated     Controls   Difference         S.E.   T-stat
----------------------------+-----------------------------------------------------------
            fte0  Unmatched | 17.0692675   20.1733333  -3.10406582   1.21654594    -2.55
                        ATT | 17.0692675   20.4710986  -3.40183108    1.4673544    -2.32
                        ATU | 20.1733333   17.6299058  -2.54342757            .        .
                        ATE |                          -3.23632912            .        .
----------------------------+-----------------------------------------------------------
            fte1  Unmatched | 17.5183121        17.65  -.131687898   1.11242821    -0.12
                        ATT | 17.5183121   17.7918742  -.273562085   1.05193873    -0.26
                        ATU |      17.65   18.0632041   .413204067            .        .
                        ATE |                          -.141152158            .        .
----------------------------+-----------------------------------------------------------
         did_fte  Unmatched | .449044586  -2.52333333   2.97237792    1.1318176     2.63
                        ATT | .449044586  -2.67922441     3.128269   1.35190664     2.31
                        ATU |-2.52333333   .433298308   2.95663164            .        .
                        ATE |                           3.09517696            .        .
----------------------------+-----------------------------------------------------------
Note: S.E. does not take into account that the propensity score is estimated.

           | psmatch2:
 psmatch2: |   Common
 Treatment |  support
assignment | On suppor |     Total
-----------+-----------+----------
 Untreated |        75 |        75 
   Treated |       314 |       314 
-----------+-----------+----------
     Total |       389 |       389 


. 
. /* (4) IPW Matching ATT=ATET and ATE */
. teffects ipw (did_fte) (treated bk kfc roys, probit), atet vce(robust) // osample(ipw1)  

Iteration 0:   EE criterion =  1.142e-28  
Iteration 1:   EE criterion =  3.433e-31  

Treatment-effects estimation                    Number of obs     =        389
Estimator      : inverse-probability weights
Outcome model  : weighted mean
Treatment model: probit
------------------------------------------------------------------------------
             |               Robust
     did_fte |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
ATET         |
     treated |
 (NJ vs PA)  |   2.631046   1.220372     2.16   0.031     .2391604    5.022931
-------------+----------------------------------------------------------------
POmean       |
     treated |
         PA  |  -2.182001   1.130429    -1.93   0.054      -4.3976    .0335979
------------------------------------------------------------------------------

. teffects ipw (did_fte) (treated bk kfc roys, probit), ate vce(robust) // osample(ipw2)  

Iteration 0:   EE criterion =  1.147e-28  
Iteration 1:   EE criterion =  1.002e-31  

Treatment-effects estimation                    Number of obs     =        389
Estimator      : inverse-probability weights
Outcome model  : weighted mean
Treatment model: probit
------------------------------------------------------------------------------
             |               Robust
     did_fte |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
ATE          |
     treated |
 (NJ vs PA)  |   2.712311   1.238966     2.19   0.029      .283982     5.14064
-------------+----------------------------------------------------------------
POmean       |
     treated |
         PA  |  -2.247811   1.146997    -1.96   0.050    -4.495884    .0002625
------------------------------------------------------------------------------

To summarize, Kernel PSM DID and its regression analogue as well as PSM with psmatch2 on differences yield an ATT of 3.13 FTEs. The weights from this program are more involved than the IPW weights.

The IPW on differences and the regression analogue you propose are similar to each other with an ATT of 2.63 FTEs, but smaller than the PSM based-estimates, though the differences are not significant.


Code:

cls
set more off
estimates clear
use http://fmwww.bc.edu/repec/bocode/c/CardKrueger1994.dta, clear
/* Need to balance the panels so that all the methods use the same sample */
drop if missing(fte)
bys id (t): keep if _N==2

/* (1) -diff- to get ATT */
diff fte, treated(treated) period(t) id(id) cov(bk kfc roys) kernel ktype(gaussian) support bw(0.05) cluster(id) robust // report
assert _support==1

/* (2a) Regression Using Kernel Weights for ATE */
reg fte i.treated##i.t [aw=_weights], cluster(id) robust
xtset id t
xtreg fte i.treated##i.t [aw=_weights], fe cluster(id) robust

/* Create ATT and ATE Weights for regressions*/
/* You can install -xfill- by typing net from http://www.sealedenvelope.com/ and clicking on the name */
xfill _ps, i(id)
gen double w_att = cond(treated==1,1,_ps/(1-_ps))
gen double w_ate = cond(treated==1,1/_ps,1/(1-_ps))

/* (2b) Regression Using ATT and ATE IPW Weights */
reg fte i.treated##i.t [pw=w_att] /*if _support==1*/, cluster(id) robust
reg fte i.treated##i.t [pw=w_ate] /*if _support==1*/, cluster(id) robust
xtreg fte i.treated##i.t [pw=w_att], fe cluster(id) robust
xtreg fte i.treated##i.t [pw=w_ate], fe cluster(id) robust

/* (3) psmatch2 for ATE and ATT */
keep id fte treated t bk kfc roys
qui reshape wide fte, i(id treated bk kfc roys) j(t)
gen did_fte = fte1 - fte0
psmatch2 treated bk kfc roys, outcome(fte0 fte1 did_fte) kernel kerneltype(normal) bw(0.05) common ate

/* (4) IPW Matching ATT=ATET and ATE */
teffects ipw (did_fte) (treated bk kfc roys, probit), atet vce(robust) // osample(ipw1)  
teffects ipw (did_fte) (treated bk kfc roys, probit), ate vce(robust) // osample(ipw2)  
$\endgroup$
  • $\begingroup$ Thank you Dimitriy, your explanation is comprehensive and very useful. Anyway, I have some questions regarding your answers above: 1. do I need to use cluster(id) option when using -diff- command? what if not? 2. You kernel regression using command -reg- even the data you used is panel, can we use xtreg instead of reg? 3. Did you use regression from (1) to (4) simultaneously? Thank you. $\endgroup$ – putut purwandono Aug 6 '16 at 12:50
  • $\begingroup$ @pututpurwandono (1) You don't have to cluster with diff, but I wanted to the standard errors to match reg, where I clustered. You could also cluster on other variables, like city or region, as long as there are enough of them. $\endgroup$ – Dimitriy V. Masterov Aug 6 '16 at 18:07
  • $\begingroup$ (2) You could certainly use xtreg, but the estimates would be identical to reg. $\endgroup$ – Dimitriy V. Masterov Aug 6 '16 at 18:08
  • $\begingroup$ (3) I am not sure what this means, but I think the answer is yes. I will add my code so that you can run the code yourself. $\endgroup$ – Dimitriy V. Masterov Aug 6 '16 at 18:09
  • 1
    $\begingroup$ You should ask another question rather than doing this in the comments. $\endgroup$ – Dimitriy V. Masterov Aug 20 '16 at 10:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.