Although my graduate training is in econometrics (ABD FTW!), I spent a few years doing more heavily IT-related work before -- now -- transitioning to a data science-y role. Questions about using applied machine learning methods to forecast and make inferences with longitudinal data have been asked here before, but I feel like my half-remembered econometrics training is muddying my thinking and I'm hoping someone could point me in the right conceptual direction.

Let's say I'm in retail and I'm trying to estimate the weekly return (say, in dollars $S_t$) for each store in terms of compliance ($C_t$) with some rule. My inclination is to think in terms of an equation like this:

$S_{it} = f(S_{it-1}, X_i, Z_{it}, C_{it}) + u_i + \epsilon_{it}$

where $X_i$ are time invariant store-level covariates, $Z_{it}$ are time varying covariates, and $u_i$ is a store-level random effect. Assume I'd like to use some non or minimally parametric method like a random forest.

1) Ignoring issues related to using the correct samples in cross validation, in such a scenario is it "enough" to run the model using a lagged outcome variable? Should $u_i$ be accounted for via a cluster-specific indicator? If so, should I be concerned about correlation with the error term as I might with a parametric linear model?

2) Assume for the sake of argument that $S_{it-1}$ plausibly has no causal impact on $S_{it}$. If my goal is inference on $C_{it}$, should I be concerned that adding a lagged outcome variable might spuriously soak up variation?

3) In general, is there some other way to take advantage of the longitudinal structure of the data here?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.