Comparing coefficients across segmented regression models given aggregated, heterogenous data

Question

My goal is to compare $\beta_1$ across $7$ models:

\begin{align*} Y^1_t &= \beta_0 + \beta_1 X^1_t + \epsilon_t \\ Y^2_t &= \beta_0 + \beta_1 X^1_t + \epsilon_t \\ &\vdots \\ Y^7_t &= \beta_0 + \beta_1 X^7_t + \epsilon_t, \end{align*} where the realizations of $Y^L_t$ is time series data and $X^L_t$ is a binary variable for $L=1, 2, \dotsc, 7$, interpreted as a treatment.

I want to compare $\beta_1$ across models to determine if the treatment had a heterogenous effect on $Y^L_t$. To be concrete, the response variables are different types of crime.

The problem is the means and variances of $Y^1, Y^2, \dotsc, Y^7$ vary greatly. For the sake of argument, suppose $Avg(Y^1) = 100, Avg(Y^2) = 4$, and $\hat{\beta^1} = -25, \hat{\beta^2} = -1$. In terms of averages, the effect is the "same", but the coefficients are quite different. So a direct comparison becomes difficult if not untenable.

I have tried to identify some approaches to compare the treatment effects:

1. Percent change

Using the pre-treatment average of $Y^L$, compute percent change $PC = \frac{\beta_1}{Avg(Y^L)} \cdot 100$, and compare those.

2. Standardize $Y^L$

Using overall Avg / Std, or the pre-treatment Avg / Std, estimate the coefficients and make direct comparisons (perhaps using a hypothesis test of their difference).

3. $Z$-test for difference of coefficients (or some other test)

4. Seemingly unrelated regression

Ideally, I want to determine if the difference in effects is statistically significant or not.

Answer 1

If willing to reparametrize the model, you might consider Poisson regression. So for Poisson regression we have:

$$\mathbb{E}[Y_t^1] = \exp(\beta_{01} + \beta_{11}X_t^1)$$ $$\mathbb{E}[Y_t^2] = \exp(\beta_{02} + \beta_{12}X_t^2)$$

etc. for your stacked equations. Note that $\exp(\beta_{01} + \beta_{11}X_t^1) = \exp(\beta_{01}) \cdot \exp(\beta_{11}X_t^1)$, and so in this formulation you get your proportional interpretation of the effects for $\beta_{11}$, $\beta_{12}$. (But I would still test for equality on the linear scale.)

For this simple of a model with all binary inputs on the right hand side, each of them will estimate the same expected value on the left hand side, it just changes the parameters and the nature of how you generalize across equations.

Stata has a program to do seemingly unrelated regression for glm's, see the suest command. Another way to do it though is to stack the equations, fit a single model, and do a likelihood ratio test for the restricted vs the model allowing the treatment effects to vary across the equations. Example below in R:

# Simulating two crime types
set.seed(10)
n <- 1000
t <- rep(0:1,n/2)
# crime type 1, ~10
l1 <- exp(2 + 0.5*t) 
c1 <- rpois(n,l1)
# crime type 2, ~130
l2 <- exp(4.6 + 0.5*t)
c2 <- rpois(n,l2)
# Prepping data
ctype <- rep(c("c1","c2"),each=n)
data <- data.frame(crime=c(c1,c2),treat=c(t,t),c=ctype)

# Linear regression
linmod <- lm(crime ~ c + treat:c - 1, data)
summary(linmod)

# Poisson regression
pmod <- glm(crime ~ c + treat:c - 1, family='poisson', data)
summary(pmod)

# Showing linmod and pmod give the same marginal predictions!
all.equal(predict(linmod),predict(pmod,type='response'))

# Restricted model with no treatment het
pmod_restrict <- glm(crime ~ c + treat - 1, family='poisson', data)
anova(pmod_restrict,pmod,test='LRT') #Fail to reject null