Picking a model: Diagnostics or Model Strengths

Question

I am building a lot of models and want to pick one to use for predicting. I am using linear regression, elastic net, and partial least squares regression.

I know my data is highly correlated and that elastic net and pls are good models to deal with collinearity. Even knowing linear regression isn't, I built models of all 3 types and compared them with a few diagnostics (on a validation dataset) . To my surprise the linear regression models are out performing the elastic net and pls models (beyond the model assumptions matching I had done a similar exercise in the past and the PLS model was the best on similar data).

My question: Is it more important to go with a model's strengths, EN and PLS are better for collinearity so I should pick one of those since I know the data is highly correlated, or is it true that if the models strengths were good enough for the data it would be picked up in the diagnostics (prediction error and RMSE being the two I am focusing on the most) so I should just stick with linear regression?

Answer 1

Multicollinearity is more of a concern when your focus is inference rather than prediction.

Since your goal is prediction, the best test of a model is how it performs on new data. So whatever the theoretical strengths and weaknesses of different modelling approaches, I would focus on their performance alone.

Of course, it can be argued that a single holdout dataset is insufficient to have confidence in the difference in performance. If the performance difference persists when measured using k-fold cross-validation, it would be more convincing evidence.