Choice of a less true model over the truer model if it predicts better and my purpose is prediction

Question

Sometimes a less true model predicts better than a truer model (When will a less true model predict better than a truer model?). So should I choose a less true model over the truer model if it predicts better and my purpose is prediction? Similarly, should I choose a model which violates the assumptions (e.g normality or homoscedasticity of the residuals in linear regression) over the model which satisfies all of the assumptions if it predicts better and my purpose is prediction?

Answer 1

I believe it depends. Do you believe/have evidence to believe that your sample represents the population that you would wish to predict? More often than not a model may fit very well with your sample but not with other samples from the same population.

If your purpose is prediction, you might wish to look at cross-validation results. If your outcome is dichotamous/categorical, you should factor in the cost of making a false positive or a false negative into your decision. (ROC plots would be useful)

Answer 2

It is very common to use a simple model that you know isn't true because the more complex model that is more realistic cannot be reliably estimated from the finite amount of training data that you actually have. Common examples include Naive Bayes (NB) for classifying documents (I would hope that nobody writes documents by drawing words at random from some distribution, although with some blogs it is hard to be absolutely certain! ;o). The use of linear models where it is likely that there is some underlying non-linearity is another common example (which is why users of neural nets are often advised to also try a linear model as a baseline). If we had infinite training data, and computational power, I would say to always use the more true model, but we are never in that situation.

The key is to be sure that the less true model reliably gives better predictions than the truer model (due to difficulty in estimation), you need to be sure that the difference is not due to things like the partitioning of the data to form test and training splits etc. If you are only interested in prediction and not understanding the data, then unbiased and low variance performance estimation will tell you the best option.

One other thing to consider is whether there is any covariate shift, i.e. the operational data do not come from the same distribution as the training data. In that case having a true causal model is likely to give better predictions than one that merely predicts well on the data available for training and testing.

Answer 3

In my opinion, the answer is 'yes, use the simpler but better predictive model'.

I assume when you say it is better at prediction, you have done things like cross-validation (or other out-of-sample prediction), or at least AIC.

The fact that the simpler model violates every single iid N(0,1) assumptions is not a problem at all, if your goal is prediction. When those assumptions are violated, your standard error of the parameters s.e.($\hat{\beta}$) are likely to be wrong. This will affect things like hypothesis testing (whether true $\beta=0$) However, why would you care about these hypothesis tests? If your goal is prediction, the model should be built (i.e. covariates should be included) base on their ability to improve prediction, not whether their coefficients are significantly different from 0. And if you are reporting, say, cross-validated prediction error, these errors will remain valid even if the residuals fail all tests.

However, because the standard errors will be wrong, I will advise not reporting them in your final write-up. This might be difficult because reviewers might not like it. But even if you report them, you should warn that they are likely wrong.

Finally, re Dikran, even if the future data comes from a different distribution as the training data, it is still quite difficult to say whether a truer causal model will predict better than a simpler model. It really depends on whether the future data is closer (albeit different) to the training data, or to the 'true' data generation process as specified in the 'true' causal model. (I have done a simulation study on this question and am in the process of writing it up.)