This is similar to other questions on leakage (for example, this post), but all of my data are generated with look-back features, and nothing can be assumed to be iid. I'm curious if you think there is information leakage or if there are any drawbacks to the following model scheme:
In the image above, the block on the left is my training data. It contains feature data (designated $x_{j_{t}}$ and target variable (designated $y_{t}$) for days 0 through 5 - let's call this the base training set. Each $x$ feature is comprised of a window function over strictly past values of itself (they're "rolling" features). Using $x_{j_{t}}$ and $y_{t}$ data, I fit a model (let's call this the level-1 model or "chained model") and get training set predictions $y_{h_{t}}$. Using $x_{j_{t}}$ and $y_{h_{t}}$, I train a "meta-model."
Technically, this is a stacked model, no? Together with the predictions on the base training set from the level-1 model, the raw features $x_{j_{t}}$ are passed through an "identity" model directly to the meta-model. My test set is comprised of feature data from days 6 and 7, as well as predictions from the level-1 model for days 6 and 7. If performance on the test set is better for my stacked model in comparison to a model trained only on the base training set, is this legitimate?
I'm skeptical that this is a valid approach, because in the level-1 model's training set there is information leakage. For example, suppose that on day $t=5$ (in the base training set) that $x_{2_{5}} = 1$, which indicates that something "good" has happened, whereas $x_{2_{0-4}} = 0$ for all of the days 0 - 4. I'm afraid that when $x_{2}$ becomes "good", this future info causes overfit on the earlier days in the train set. But, on the other hand, I'm having difficulty seeing how gains in test set performance could be anything other than legitimate, because no data from the test set is used to train the level-1 model, and overfitting the level-1 model is, in the end, perhaps independent of the issue of information leakage into the test set.
Am I missing something here? Thanks so much!
