3
$\begingroup$

My goal is to compare $\beta_1$ across $7$ models:

\begin{align*} Y^1_t &= \beta_0 + \beta_1 X^1_t + \epsilon_t \\ Y^2_t &= \beta_0 + \beta_1 X^1_t + \epsilon_t \\ &\vdots \\ Y^7_t &= \beta_0 + \beta_1 X^7_t + \epsilon_t, \end{align*} where the realizations of $Y^L_t$ is time series data and $X^L_t$ is a binary variable for $L=1, 2, \dotsc, 7$, interpreted as a treatment.

I want to compare $\beta_1$ across models to determine if the treatment had a heterogenous effect on $Y^L_t$. To be concrete, the response variables are different types of crime.

The problem is the means and variances of $Y^1, Y^2, \dotsc, Y^7$ vary greatly. For the sake of argument, suppose $Avg(Y^1) = 100, Avg(Y^2) = 4$, and $\hat{\beta^1} = -25, \hat{\beta^2} = -1$. In terms of averages, the effect is the "same", but the coefficients are quite different. So a direct comparison becomes difficult if not untenable.

I have tried to identify some approaches to compare the treatment effects:

  1. Percent change

Using the pre-treatment average of $Y^L$, compute percent change $PC = \frac{\beta_1}{Avg(Y^L)} \cdot 100$, and compare those.

  1. Standardize $Y^L$

Using overall Avg / Std, or the pre-treatment Avg / Std, estimate the coefficients and make direct comparisons (perhaps using a hypothesis test of their difference).

  1. $Z$-test for difference of coefficients (or some other test)

  2. Seemingly unrelated regression

Ideally, I want to determine if the difference in effects is statistically significant or not.

$\endgroup$
2
$\begingroup$

If willing to reparametrize the model, you might consider Poisson regression. So for Poisson regression we have:

$$\mathbb{E}[Y_t^1] = \exp(\beta_{01} + \beta_{11}X_t^1)$$ $$\mathbb{E}[Y_t^2] = \exp(\beta_{02} + \beta_{12}X_t^2)$$

etc. for your stacked equations. Note that $\exp(\beta_{01} + \beta_{11}X_t^1) = \exp(\beta_{01}) \cdot \exp(\beta_{11}X_t^1)$, and so in this formulation you get your proportional interpretation of the effects for $\beta_{11}$, $\beta_{12}$. (But I would still test for equality on the linear scale.)

For this simple of a model with all binary inputs on the right hand side, each of them will estimate the same expected value on the left hand side, it just changes the parameters and the nature of how you generalize across equations.

Stata has a program to do seemingly unrelated regression for glm's, see the suest command. Another way to do it though is to stack the equations, fit a single model, and do a likelihood ratio test for the restricted vs the model allowing the treatment effects to vary across the equations. Example below in R:

# Simulating two crime types
set.seed(10)
n <- 1000
t <- rep(0:1,n/2)
# crime type 1, ~10
l1 <- exp(2 + 0.5*t) 
c1 <- rpois(n,l1)
# crime type 2, ~130
l2 <- exp(4.6 + 0.5*t)
c2 <- rpois(n,l2)
# Prepping data
ctype <- rep(c("c1","c2"),each=n)
data <- data.frame(crime=c(c1,c2),treat=c(t,t),c=ctype)

# Linear regression
linmod <- lm(crime ~ c + treat:c - 1, data)
summary(linmod)

# Poisson regression
pmod <- glm(crime ~ c + treat:c - 1, family='poisson', data)
summary(pmod)

# Showing linmod and pmod give the same marginal predictions!
all.equal(predict(linmod),predict(pmod,type='response'))

# Restricted model with no treatment het
pmod_restrict <- glm(crime ~ c + treat - 1, family='poisson', data)
anova(pmod_restrict,pmod,test='LRT') #Fail to reject null
$\endgroup$
5
  • $\begingroup$ Does the LRT apply to the overall difference between the restricted / unrestricted models, or does it apply to the specific coefficient? Say I have 3 regressors (the same for both models) and I restrict x1, does this test make a claim about x1? $\endgroup$
    – Andrew
    Jun 17 at 12:45
  • $\begingroup$ LRT is for the overall model - so all differences in the model at once. If you want to test individual coefficient contrasts one-by-one, doing the approach via SUR and then Wald tests makes sense. (You can do the Wald tests via the stacked model as well.) $\endgroup$
    – Andy W
    Jun 17 at 13:05
  • $\begingroup$ I'm assuming the Wald Test applies to the coefficient that is restricted? Is there a way to implement a SUR model with Wald tests in Python? I have attempted Statsmodels and linearmodels, but the former has not released a stable version of SUR, and the latter does not have Wald for SUR, to my knowledge. $\endgroup$
    – Andrew
    Jun 17 at 14:11
  • $\begingroup$ Yeah not sure how Stata implements SUR for GLM's, so if going with python would suggest the stacked model approach. For the Wald tests you should adjust the variance/covariance matrix to account for clustering (and possible overdispersion with Poisson models). See here for example Wald test in python. I believe the R matrix should sum to 0 in this example, so if doing three coefficients set two to -1/2 and the other coefficient to 1. $\endgroup$
    – Andy W
    Jun 17 at 14:50
  • 2
    $\begingroup$ Here you go @Andrew, wrote a blog post to show how to do the Wald tests in python via the stacked model approach. $\endgroup$
    – Andy W
    Jun 18 at 13:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.