# Is splitting the data into test and training sets purely a "stats" thing?

I am a physics student studying machine learning / data science, so I don't mean for this question to start any conflicts :) However, a big part of any physics undergraduate program is to do labs/experiments, which means a lot of data processing and statistical analysis. However, I notice a sharp difference between the way physicists deal with data and the way my data science / statistical learning books deal with data.

The key difference is that when trying to perform regressions to data obtained from physics experiments, the regression algorithms are applied to the WHOLE dataset, there is absolutely no splitting into training and test sets. In the physics world, the R^2 or some type of pseudo-R^2 is calculated for the model based on the whole data set. In the stats world, the data is almost always split up into 80-20, 70-30, etc... and then the model is evaluated against the test dataset.

There are also some major physics experiments (ATLAS, BICEP2, etc...) that never do this data splitting, so I'm wondering why there is a such a staunch difference between the way physicists/experimentalists do statistics and the way data scientists do statistics.

• (+1) very nice question (that I don't have the time properly answering). Comment: Physics has the luxury of "real experiments"; generally controlled/laboratory conditions, mostly well-defined outcomes/variables and assummed repeatability. Usual Public Health/Econometrics/Survey Statistics projects (to mention a few obvious sub-fields) just don't get that. Confounding, seasonality (time-dependence) and generally concept drift is rife in Statistics so this "splitting of data" is one of the obvious ways to prevent totally silly results. Plus not all estimators are created equally efficient. :) Commented Jul 1, 2017 at 23:21
• You will find a wealth of relevant discussion and background in a recent discussion paper by David Donoho, a statistics Professor at Stanford: courses.csail.mit.edu/18.337/2015/docs/50YearsDataScience.pdf See particularly the discussion of "Predictive Culture" as contrasted with traditional statistics. Commented Jul 2, 2017 at 0:07
• I think it's a "prediction in the absence of theory" thing, which is a small subset of "stats", and a large subset of machine learning. Commented Jul 2, 2017 at 0:13
• statisticians don't split their data either (p < .05) Commented Jul 2, 2017 at 22:42
• @rep_ho some -- perhaps many - statisticians involved with situations where out of sample prediction is important do so (and some have done for a long time). ideas like crossvalidation and leave-one-out statistics (for example) have been around for ages. Statisticians tend not to split just once, though, unless that's unavoidable. It may depend on which statisticians you talk to Commented Jul 2, 2017 at 23:47

Not all statistical procedures split in to training/testing data, also called "cross-validation" (although the entire procedure involves a little more than that).

Rather, this is a technique that specifically is used to estimate out-of-sample error; i.e. how well will your model predict new outcomes using a new dataset? This becomes a very important issue when you have, for example, a very large number of predictors relative to the number of samples in your dataset. In such cases, it is really easy to build a model with great in-sample error but terrible out of sample error (called "over fitting"). In the cases where you have both a large number of predictors and a large number of samples, cross-validation is a necessary tool to help assess how well the model will behave when predicting on new data. It's also an important tool when choosing between competing predictive models.

On another note, cross-validation is almost always just used when trying to build a predictive model. In general, it is not very helpful for models when you are trying to estimate the effect of some treatment. For example, if you are comparing the distribution of tensile strength between materials A and B ("treatment" being material type), cross validation will not be necessary; while we do hope that our estimate of treatment effect generalizes out of sample, for most problems classic statistical theory can answer this (i.e. "standard errors" of estimates) more precisely than cross-validation. Unfortunately, classical statistical methodology1 for standard errors doesn't hold up in the case of overfitting. Cross-validation often does much better in that case.

On the other hand, if you are trying to predict when a material will break based on 10,000 measured variables that you throw into some machine learning model based on 100,000 observations, you'll have a lot of trouble building a great model without cross validation!

I'm guessing in a lot of the physics experiments done, you are generally interested in estimation of effects. In those cases, there is very little need for cross-validation.

1One could argue that Bayesian methods with informative priors are a classical statistical methodology that addresses overfitting. But that's another discussion.

Side note: while cross-validation first appeared in the statistics literature, and is definitely used by people who call themselves statisticians, it's become a fundamental required tool in the machine learning community. Lots of stats models will work well without the use of cross-validation, but almost all models that are considered "machine learning predictive models" need cross-validation, as they often require selection of tuning parameters, which is almost impossible to do without cross-validation.

• I think that your latest edit about the prediction/inference distinction is a bit off and prone to misinterpretation (which I might do right now). If anything, if we infer that material A is stronger than B we want this to hold out-of-sample too. Plus, such an idea would ignore bootstrap/permutations approaches. And the example is a bit off; an $n \approx p$ is not really saved by a train-test-split approach but rather from regularisation. Commented Jul 1, 2017 at 23:57
• @usεr11852: yes, but it's nearly impossible to pick reasonable regularization penalties without cross-validation (other than thinking about penalties as Bayesian priors, but that's hard with black box models!). And while we do want our results in comparing A to B to hold out of sample, this typically is not a problem that requires model tuning (like prediction often does), and with the relatively low numbers of parameters, classical statistical theory can handle this without using cross validation. Commented Jul 2, 2017 at 0:52
• This is a circular argument, regularisation uses cross-validation but cross-validation is done for regularisation. That's why I commented somewhat against it to begin with. I think statistical inference/causality moves away from this non-model tuning approach (see for example 2016 Johansson et al. "Learning representations for counterfactual inference" - such a messy beautiful paper). Finally Fundamental Physics research when presented it hard problems can also rely on ML (eg. the Higgs Boson Machine Learning Challenge) approaches. Commented Jul 2, 2017 at 12:19
• @usεr11852 Regularization does not "use" cross-validation, but rather your tuning parameter for regularization is chosen using cross validation. For example, see glment's cv.glmnet for the whole procedure in a nice compact function. Commented Jul 2, 2017 at 16:52
• Also, I never made the claim that physics research cannot use ML approaches nor cross-validation! I was only explaining that cross-validation is typically used specifically for choosing between complex models/tuning parameters in predictive models, and that in many classic physics experiments, cross-validation is not necessary. So what physicists do with that data is not necessarily at odds with what statisticians would do with that data, which I believe was the core of the OP's question. Commented Jul 2, 2017 at 16:56

Being (analytical) chemist, I encounter both approaches: analytical calculation of figures of merit [mostly for univariate regression] as well as direct measurement of predictive figures of merit.
The train/test splitting to me is the "little brother" of a validation experiment to measure prediction quality.

However, in my professional life, I deal with spectroscopic data sets (typically 100s to 1000s of variates $p$) and moreover with rather limited sets of independent cases (samples) $n$. Often $n < p$, so we use regularization of which it is not always easy to say how many degrees of freedom we use, and in addition we try to at least somewhat compensate for the small $n$ by using (large) numbers of almost repeated measurements - which leaves us with an unknown effective $n$. Without knowing $n$ or $df$, the classical approaches don't work. But as I'm mostly doing predictions, I always have a very direct possibility of measuring the predictive ability of my model: I do predictions, and compare them to reference values.