Here are my personal understandings, so feel free to correct me or add comments.
Under the classical statistical modeling framework, e.g. regression, we are often facing two questions: 1) is this a correct model to apply? for instance a linear model? is the response gaussian distributed? are these the proper predictors to include? 2) how are the predictive performance of the predictors (or features). To answer these 2 questions: The selection of the correct model type is often done by maximizing the likelihood (a kind of joint distribution) of the response data, for instance using AIC, BIC. The performance of the predictors are presented as p-values/effect size, calculated from the estimated residual SD and coefficient SE, under their asymptotic distributions.
We can see that the classical frame relies heavily on the assumed distributions. However, In machine learning, the assumptions of such distributions are not addressed. There seems also not a clear separation of model selection and predictor performance concepts. Instead, models are selected by minimizing the error in an independent test data (corss-validation procedure) and the corresponding predictors performance are presented as the error from the test data, e.g. mean square error.
My questions are:
1) is my statement above correct?
2) Are these two ways of thinking fundamentally different? In what situation could the two methods do the same thing? What are their advantages and disadvantages? For instance, is the machine learning method really distribution assumption free? But the models used in machine learning are not always assumption free, some also assume feature distributions (e.g. gaussian classifiers). Is the cross-validation method superior in evaluating complicated models? How robust are both methods with small sample size? (with small sample size, the asymptotic distribution is not valid in classical method)
3) Why the cross-validation procedure is not popular yet in the classical statistical field? Still many researchs are based on likelihood and p-value. Is there a reason that it is not necessary to adopt it.e.g. hypothesis testing?