Is splitting the data into test and training sets purely a "stats" thing?

Question

I am a physics student studying machine learning / data science, so I don't mean for this question to start any conflicts :) However, a big part of any physics undergraduate program is to do labs/experiments, which means a lot of data processing and statistical analysis. However, I notice a sharp difference between the way physicists deal with data and the way my data science / statistical learning books deal with data.

The key difference is that when trying to perform regressions to data obtained from physics experiments, the regression algorithms are applied to the WHOLE dataset, there is absolutely no splitting into training and test sets. In the physics world, the R^2 or some type of pseudo-R^2 is calculated for the model based on the whole data set. In the stats world, the data is almost always split up into 80-20, 70-30, etc... and then the model is evaluated against the test dataset.

There are also some major physics experiments (ATLAS, BICEP2, etc...) that never do this data splitting, so I'm wondering why there is a such a staunch difference between the way physicists/experimentalists do statistics and the way data scientists do statistics.

Answer 1

Not all statistical procedures split in to training/testing data, also called "cross-validation" (although the entire procedure involves a little more than that).

Rather, this is a technique that specifically is used to estimate out-of-sample error; i.e. how well will your model predict new outcomes using a new dataset? This becomes a very important issue when you have, for example, a very large number of predictors relative to the number of samples in your dataset. In such cases, it is really easy to build a model with great in-sample error but terrible out of sample error (called "over fitting"). In the cases where you have both a large number of predictors and a large number of samples, cross-validation is a necessary tool to help assess how well the model will behave when predicting on new data. It's also an important tool when choosing between competing predictive models.

On another note, cross-validation is almost always just used when trying to build a predictive model. In general, it is not very helpful for models when you are trying to estimate the effect of some treatment. For example, if you are comparing the distribution of tensile strength between materials A and B ("treatment" being material type), cross validation will not be necessary; while we do hope that our estimate of treatment effect generalizes out of sample, for most problems classic statistical theory can answer this (i.e. "standard errors" of estimates) more precisely than cross-validation. Unfortunately, classical statistical methodology¹ for standard errors doesn't hold up in the case of overfitting. Cross-validation often does much better in that case.

On the other hand, if you are trying to predict when a material will break based on 10,000 measured variables that you throw into some machine learning model based on 100,000 observations, you'll have a lot of trouble building a great model without cross validation!

I'm guessing in a lot of the physics experiments done, you are generally interested in estimation of effects. In those cases, there is very little need for cross-validation.

¹One could argue that Bayesian methods with informative priors are a classical statistical methodology that addresses overfitting. But that's another discussion.

Side note: while cross-validation first appeared in the statistics literature, and is definitely used by people who call themselves statisticians, it's become a fundamental required tool in the machine learning community. Lots of stats models will work well without the use of cross-validation, but almost all models that are considered "machine learning predictive models" need cross-validation, as they often require selection of tuning parameters, which is almost impossible to do without cross-validation.

Answer 2

Being (analytical) chemist, I encounter both approaches: analytical calculation of figures of merit [mostly for univariate regression] as well as direct measurement of predictive figures of merit.
The train/test splitting to me is the "little brother" of a validation experiment to measure prediction quality.

---

Long answer:

The typical experiments we do e.g. in undergraduate physical chemistry use univariate regression. The property of interest are often the model parameters, e.g. the time constant when measuring reaction kinetics, but sometimes also predictions (e.g. univariate linear calibration to predict/measure some value of interest).
These situations are very benign in terms of not overfitting: there's usually a comfortable number of degrees of freedom left after all parameters are estimated, and they are used to train (as in education) students with classical confidence or prediction interval calculation, and classical error propagation - they were developed for these situations. And even if the situation is not entirely textbook-like (e.g. I have structure in my data, e.g. in the kinetics I'd expect the data is better described by variance between runs of the reaction + variance between measurements in a run than by a plain one-variance-only approach), I can typically have enough runs of the experiment to still get useful results.

However, in my professional life, I deal with spectroscopic data sets (typically 100s to 1000s of variates $p$) and moreover with rather limited sets of independent cases (samples) $n$. Often $n < p$, so we use regularization of which it is not always easy to say how many degrees of freedom we use, and in addition we try to at least somewhat compensate for the small $n$ by using (large) numbers of almost repeated measurements - which leaves us with an unknown effective $n$. Without knowing $n$ or $df$, the classical approaches don't work. But as I'm mostly doing predictions, I always have a very direct possibility of measuring the predictive ability of my model: I do predictions, and compare them to reference values.

This approach is actually very powerful (though costly due to increased experimental effort), as it allows me to probe predictive quality also for conditions that were not covered in the training/calibration data. E.g. I can measure how predictive quality deteriorates with extrapolation (extrapolation includes also e.g. measurements made, say, a month after the training data was acquired), I can probe the ruggedness against confounding factors that I expect to be important, etc. In other words, we can study the behaviour of our model just as we study the behavior of any other system: we probe certain points, or perturb it and look at the change in the system's answer, etc.

I'd say that the more important predictive quality is (and the higher the risk of overfitting) the more we tend to prefer direct measurements of predictive quality rather than analytically derived numbers. (Of course we could have included all those confounders also into the design of the training experiment). Some areas such as medical diagnostics demand that proper validation studies are performed before the model is "let loose" on real patients.

The train/test splitting (whether hold out* or cross validation or out-of-bootstrap or ...) takes this one step easier. We save the extra experiment and do not extrapolate (we only generalize to predicting unknown independent cases of the very same distribution of the training data). I'd describe this as a verification rather than validation (although validation is deeply in the terminology here). This is often the pragmatic way to go if there are not too high demands on the precision of the figures of merit (they may not need to be known very precisely in a proof-of-concept scenario).

* do not confuse a single random split into train and test with a properly designed study to measure prediction quality.