The following scenario must be extremely common but I couldn't find a best practice for it readily available.
Suppose we require a predictive model (m) that is supposed to explain some variable y based on data X: $ m X = y $
Now, let's say we don't have any appropriate validation data to us available. All we can do is define some proxy $y'$ that is supposedly a good substitute for what we need to predict.
Then we run an experiment over time. For simplicity we assume X, y don't vary as function of time. Each timestep we have more (but few) data coming in so that we have some way of validating out model.
My questions are,
1: how do we get the most optimal prediction at each timestep?
2: any prediction on y' would be biased and at the start anything based on y does not have sufficient amount of data, but at what point would a combination work?
3: Is there a way to make a single prediction based on both inputs or would you have to make 2 predictions and do some ensemble afterwards? Over time you would downweigh the initial prediction based on the proxy data but how to compute these weights properly? I Suspect practically speaking this would be annoying to automatize having to recreate data models and retraining every timestep.
Ideally of course you would just have a working model that improves automatically given some new input. (Not sure if LSTM would be suitable in my application, both the total nr of samples and those incoming every timestep we'd be working with is relatively low)