Is test MSE all that matters when it comes to prediction? I'm wondering if when it comes to predictive analytics, whether a lower test MSE is really all that matters. Should I not even look at residuals - other model diagnostics when it comes to prediction if I'm getting a lower test MSE?
Or does a lower test MSE model always satisfy the model assumptions? at least more than a higher test MSE model?
 A: Short answer: it will all depend on your loss-functions.
For instance, I do forecasting for replenishment in retail stores, like supermarkets or drugstores. The only forecast that is important in this context is a high quantile, because we don't want to replenish based on average demand, but to attain a high service level. (This is not always easy to communicate to our clients.) Other use cases may indeed be most interested in getting the expectation right, so the MSE would indeed be useful (but see below).
You may want to look at the tag wikis for some prediction error measures, like the mae, mape or mase, all of which are minimized in expectation for different functionals of the true future distribution.
And of course you should still look at your errors to see whether there is any remaining structure you could exploit to improve your algorithms or models. Method A may have a lower MSE than method B on average - but it may completely (and unacceptably) break down in certain rare cases. In such a case, you want to investigate why it breaks down in such cases and hopefully improve it. (I have found that not only looking at average errors, but also at high quantiles of errors is often helpful.)
Finally, of course, if you are not predicting numerically, but classifying, then MSE is pretty much meaningless, and you need to look into confusion matrices and base your [tags:loss-functions] on false positive and false negative rates and similar concepts.
A: If all you care about is predictive performance and your task is to select one model out of several alternatives, then it makes sense to pick the one with the lowest test error. But, modeling can be an iterative process, and you might learn something useful by looking at other aspects of a model's behavior. What you learn might help you formulate a new model with even better predictive performance.
For example, you mentioned looking at the residuals. Imagine a regression task, where you want to predict $y$ given $x$. Say that $y$ is a quadratic function of $x$, but you don't know this ahead of time. As a first guess, you fit a linear model. You could fit a number of linear models and pick the one with lowest test error. But, predictive performance would still be sub-optimal because you have the wrong model class. The MSE alone can't tell you this because it doesn't distinguish between error due to 'noise' vs. error due to unmodeled 'structure'. If you looked at the residuals (e.g. as a function of predicted $y$ value), you could see remaining structure that the linear model hasn't accounted for. At the very least, this would tell you to try a more complicated model. Hopefully, it could give a hint about the nature of the model to try next. Of course, you'd have to be careful not to overfit by repeatedly peeking at the data.
