2
$\begingroup$

A practicing statistician I know advocates strongly against cross-validation, claiming that he would rather build the model on the entire dataset and make sure the underlying statistical assumptions are rock solid, as opposed to cross-validating his models. With solid statistical foundations, he argues the models should predict well out-of-sample.

What are your arguments for or against cross-validation?

$\endgroup$
  • $\begingroup$ How would your friend work with models that have parameters that are not directly estimable from the data? (SVM, random forest, neural networks...) $\endgroup$ – Sycorax Oct 1 '16 at 22:40
  • $\begingroup$ I should have clarified before: he is a very traditional, self-labeled "dinosaur" statistician who sticks mostly to logit and multiple regression, maybe adventuring into a random forest every now and then. So, for the purposes of this question, forget about those more advanced models. $\endgroup$ – blacksite Oct 1 '16 at 22:48
  • $\begingroup$ Your friend is a data analyst not a machine learning person. $\endgroup$ – SmallChess Oct 1 '16 at 22:58
  • $\begingroup$ It's ok never cv if the job is about regression. $\endgroup$ – SmallChess Oct 1 '16 at 22:59
  • $\begingroup$ Logistic and other forms of regression are ML methods, regardless of the fact they're perceived as less "advanced" than neural nets, SVMs, etc. This isn't a question of job-title semantics. Do you have any resources as to why not cross-validating is "ok" with regression? $\endgroup$ – blacksite Oct 1 '16 at 23:45
2
$\begingroup$

rather build the model on the entire dataset

That is what is usually done for cross validation (in the validation sense): the test results for the surrogate models are used as approximation for prediction of unknown cases of the model built on the entire dataset.

and make sure the underlying statistical assumptions are rock solid,

This is (obviously) always a good idea. But it is not opposed to any kind of validation method.

as opposed to cross-validating his models.

Here's the weird thing: both points above are not opposed to cross validation.


So: after building the model on the entire data set and having made sure the modeling assumptions are met, how does your statistician friend validate the model?


Note that there are situations where cross validation is not a good choice, e.g. if you have multiple important confounders so you need to split independently of all of them, you end up with very small test and training set sizes for the surrogate models.

$\endgroup$
5
$\begingroup$

With solid statistical foundations, he argues the models should predict well out-of-sample.

Assuming that by "the underlying statistical assumptions are rock-solid" and "solid statistical foundations" he means "accurate assumptions", this is wrong. Models with more accurate assumptions may make worse out-of-sample predictions. Domingos and Pazzani (1997) has an example of this: even with simulated data generated from a known, complex model, a simpler model can perform better than the true model.

Domingos, P., & Pazzani, M. (1997). On the optimality of the simple Bayesian classifier under zero-one loss. Machine Learning, 29, 103–130. doi:10.1023/A:1007413511361

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.