A lot of people have excellent answers, here is my $0.02.
There are two ways to look at "best model", or "model selection", speaking statistically:
1 An explanation that is as simple as
possible, but no simpler (Attrib.
- This is also called Occam's Razor, as explanation applies here.
- Have a concept of True model or a model which approximates the truth
- Explanation is like doing scientific research
2 Prediction is the interest, similar to engineering development.
- Prediction is the aim, and all that matters is that the model works
- Model choice should be based on quality of predictions
- Cf: Ein-Dor, P. & Feldmesser, J. (1987) Attributes of the performance of central processing units: a relative performance prediction model. Communications of the ACM 30, 308–317.
Model Choice is equivalent to choosing the best model
For explanation we ought to be alert to be possibility of there being several
(roughly) equally good explanatory models. Simplicity helps both with communicating the concepts embodied in the model and in what psychologists call generalization, the ability to ‘work’ in scenarios very different from those in which the model was studied. So
there is a premium on few models.
For prediction: (Dr Ripley's) good analogy is that of choosing between expert
opinions: if you have access to a large panel of experts, how would you
use their opinions?
Cross Validation takes care of the prediction aspect. For details about CV please refer to this presentation by Dr. B. D. Ripley Dr. Brian D. Ripley's presentation on model selection
Citation: Please note that everything in this answer is from the presentation cited above. I am a big fan of this presentation and I like it. Other opinions may vary. The title of the presentation is: "Selecting Amongst Large Classes of Models" and was given at Symposium in Honour of John Nelder's 80th birthday, Imperial College, 29/30 March 2004, by Dr. Brian D. Ripley.