I hope this is the right place to ask this question. But if it is not, please feel free to migrate. There is a famous quote, which is like "all models are wrong, but a few are useful". So, I was just wondering, what is the influence of the model selection in a practical project? For instance, suppose we have two models, one is statistically more precise than the other one. So, the more preciser one should be used instead of the not so precise one? Is this true? Many thanks for your time and attention.
$\begingroup$
$\endgroup$
1
-
1$\begingroup$ I think the exact phrase is: "all models are wrong, but some are useful" by George E. P. Box. $\endgroup$– usεr11852Commented Sep 15, 2015 at 18:11
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
The negative influence of model selection in a particular project is that it can lead to data-dredging where you essentially over-fit and draw wrong conclusions from your data. An excellent post on the matter by the user @gung can be found here.
The positive influence of model selection is that it can allow you to test if particular covariates (or modelling assumption in general) are of statistical significance, given your data.
It is not universally true that the "more precise (model) one should used". While a standard goodness-of-fit measurement (eg. Residual-Sum-of-Squares) can be informative on first instance there are dozens other criteria that might employed (eg. Akaike information criterion) for further analysis. If you need to do some sort of model selection I would urge to always consider to resampling approaches(eg. Cross-validation) to make sure that what you observe is not an artefact of your particular dataset and/or utilise a modelling approach that explicitly penalises model complexity (eg. LASSO).
$\begingroup$
$\endgroup$
You should pick the model that performs best on the your holdout sample. Your holdout set reduces the risk of overfitting. The model that is more precise on your training set could be the model that is overfit. (What's a real-world example of "overfitting"?)
In other words, you should pick the model that will have the lowest prediction error on future data when your project is running in production. I recommend Chapter 7 of "The Elements of Statistical Learning", a free but excellent book by famous Stanford machine learning experts. http://web.stanford.edu/~hastie/local.ftp/Springer/OLD/ESLII_print4.pdf
Ultimately it depends on your use case. You tagged this question as a 'machine learning' question, so I'm assuming your use case considers predictive power to be more important than explanatory power.
answered May 29, 2015 at 1:27