Interpreting predictive models in the presence of omitted variables

Question

Suppose the best predictive model from a set of possible models is a univariable model, due to lots of moderate correlations with other variables for example. However, if I use this model for predictions of the future using new values of the single variable at a particular location, then I am implicitly assuming that all the other originally excluded covariates change according to their covariances with the included variable in the original set. However, it strikes me that this is likely to be unreasonable in many cases, e.g. if the included predictive variable is related to human activity, but the original moderate correlations caused physical variables like climate to be excluded from the predictive model. My question is, is my interpretation/exposition of this example correct?

Answer 1

Any time you are making an inference or prediction out of sample, you are assuming that the sample you have analysed is representative. That means - among other things - that the covariance structure within your sample is approximately the same as the covariance structure in the population. If that is untrue, then your model inferences & predictions are going to be flawed.

As you note, this is a big challenge with time series analysis, where covariances frequently change. However, there is an important difference in your example: you are referring to variables that have been omitted from the model. Presumably you have omitted these variables because they do not contribute much in terms of predictive power. As long as their covariances with the target/dependent variable does not change (i.e. they continue to be unimportant predictors), the covariance with the predictor you are interested in does not matter.

For example, suppose I always eat ice cream after Italian food when it is hot. You write a model that predicts when I will eat ice cream based on the temperature but leave out the Italian food. In the future, it will not matter if I also start eating ice cream also after Thai food, as long as I'm still only eating the ice cream when it is hot. The covariance between temperature and cuisine does not matter. The relationship you have modelled (ice cream consumption as a function of temperature) has remained stable and your predictions should be fine.