2
$\begingroup$

This is a statistical question, but more of a meta-statistical question. However, it's not a meta-discussion question, so I hope this is the correct venue.

I'm asking this because maybe I'm just not understanding something. I recently did a mixed-level GLM to investigate potential effects of a binary predictor on a continuous metric, over several years, for several locations. I set up a two-way interaction model with random intercepts and slopes per location. A classic repeat-measures mixed-level model.

Someone pipes up that such a model merely showed "correlation" and not "causation" and said that the question would only be worth looking at with a differences-of-differences (DID) model. Basically a "come back when you're ready to play" response.

Here's the thing that's got my head scratching. A DID is essentially a type of regression model, where "event" and "time" are binary coded and used as predictors (with interaction) vs. a response variable. Am I right? The basic DID also doesn't have compensation for autocorrelation.

I'm scratching my head, because I don't see how DID shows causation vs. correlation any more than any other type of model.

Am I just somehow not seeing something very obvious?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

A random intercept-random slopes model (which I'll also call a growth model, and which has also been called a multilevel model, a heirarchical model, and so on) does not formally evaluate difference-in-difference. "Difference-in-difference" is a widely used concept to refer to departures from previously established trends. Being unaware of the design in consideration, I cannot comment on the relevance of this person's critique of your modeling strategy. However, it leads me to guess that there must be some "pre-post" aspect to the design, where you are using groups as their own historical controls. If that's the case, your model (as stated) does not distinguish between the two individual level time-periods; the random slopes can be seen as being in the gross aggregate, they borrow trends across time which were established during the historical control period and which were modified following the occurrence of "event".

If you fit a simple segmented regression model, a difference-in-difference could be formally evaluated by considering adjustment for time, event, and their interaction with fixed effects. Modeling assumptions (independence and homoscedasticity) could be relaxed by using robust standard errors to obtain confidence intervals and p-values. A model-based approach using a growth model is to model two times: pre-event and post-event, then use a paired t-test for mean differences in the random slopes for those times either in a mixed model or structural equation model.

As a last note, causation can only be showed when the appropriate criteria for causation are met. In addition to choosing an appropriate model to the design (not clear at this point), this also depends on other factors as well, such as whether event was randomly allocated or not, whether Hawthorne effect is appropriately controlled by an historical control design, and so on. Again, reading into the comment a bit, I sense the critic was asking for a "formal" evaluation of growth, or difference-in-difference.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks for the explanation. Oddly enough, though, when I ran the DID, what did I get? Same results: Time wasn't significant. Event was significant. Interaction of time and event (DID) was not significant. All terms (time, event, and DID/interaction) also were in the same positive or negative direction of effect, too. Does that mean that the mixed-level model wasn't worth looking at? $\endgroup$
    – Bryan
    Commented Oct 25, 2017 at 12:18
  • $\begingroup$ @Bryan I'd hazard you not to read too much into this "significance" business, it's misleading. Each model summarizes data with an imposed set of assumptions. Focus instead on describing those assumptions and where possible use strategies to reduce assumptions (via more general models or inference) or check them. When you say you "ran the DID" it sounds like you fit a simple linear regression with linear, continuous time adjustment and used model-based errors for inference. Do you feel confident that this sufficiently explained trends insofar as the residuals are uncorrelated? I wouldn't. $\endgroup$
    – AdamO
    Commented Oct 25, 2017 at 17:38
  • $\begingroup$ @Bryan I described some other methods of data analysis for this type of problem. It sounds like you may benefit from doing deeper (self-directed) research on longitudinal modeling, growth models, sandwich standard errors, and so on. $\endgroup$
    – AdamO
    Commented Oct 25, 2017 at 17:39
  • 1
    $\begingroup$ I did not do what you claim I did. I coded the time as 0 = before exposure, 1 = after exposure. All entities that did not undergo the event were coded as 0, all those that did were coded at 1. (Two variables, time and event). I did not NOT use a linear, continuous time adjustment. You made an enormous and utterly unwarranted assumption. $\endgroup$
    – Bryan
    Commented Oct 27, 2017 at 13:24
  • $\begingroup$ @Bryan If I'm correct in sensing some defensiveness in your language, then I'll ask you to be clearer about what you did. You say you ran difference in difference, but did not say what that was. I've read quite a literature on the subject and it's highly inconsistent. To describe your method, I would say that you modeled time as a step function. If you want to assess the adequacy of that assumption, plot the residuals versus calendar time. Rising or falling is suggestive of linear effects which you may not have captured. Also consider variograms to look at other types of autoregressive trends. $\endgroup$
    – AdamO
    Commented Oct 27, 2017 at 13:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.