9
$\begingroup$

I am a physics student studying machine learning / data science, so I don't mean for this question to start any conflicts :) However, a big part of any physics undergraduate program is to do labs/experiments, which means a lot of data processing and statistical analysis. However, I notice a sharp difference between the way physicists deal with data and the way my data science / statistical learning books deal with data.

The key difference is that when trying to perform regressions to data obtained from physics experiments, the regression algorithms are applied to the WHOLE dataset, there is absolutely no splitting into training and test sets. In the physics world, the R^2 or some type of pseudo-R^2 is calculated for the model based on the whole data set. In the stats world, the data is almost always split up into 80-20, 70-30, etc... and then the model is evaluated against the test dataset.

There are also some major physics experiments (ATLAS, BICEP2, etc...) that never do this data splitting, so I'm wondering why there is a such a staunch difference between the way physicists/experimentalists do statistics and the way data scientists do statistics.

$\endgroup$
  • 1
    $\begingroup$ (+1) very nice question (that I don't have the time properly answering). Comment: Physics has the luxury of "real experiments"; generally controlled/laboratory conditions, mostly well-defined outcomes/variables and assummed repeatability. Usual Public Health/Econometrics/Survey Statistics projects (to mention a few obvious sub-fields) just don't get that. Confounding, seasonality (time-dependence) and generally concept drift is rife in Statistics so this "splitting of data" is one of the obvious ways to prevent totally silly results. Plus not all estimators are created equally efficient. :) $\endgroup$ – usεr11852 Jul 1 '17 at 23:21
  • 3
    $\begingroup$ You will find a wealth of relevant discussion and background in a recent discussion paper by David Donoho, a statistics Professor at Stanford: courses.csail.mit.edu/18.337/2015/docs/50YearsDataScience.pdf See particularly the discussion of "Predictive Culture" as contrasted with traditional statistics. $\endgroup$ – Gordon Smyth Jul 2 '17 at 0:07
  • 1
    $\begingroup$ I think it's a "prediction in the absence of theory" thing, which is a small subset of "stats", and a large subset of machine learning. $\endgroup$ – The Laconic Jul 2 '17 at 0:13
  • $\begingroup$ statisticians don't split their data either (p < .05) $\endgroup$ – rep_ho Jul 2 '17 at 22:42
  • $\begingroup$ @rep_ho some -- perhaps many - statisticians involved with situations where out of sample prediction is important do so (and some have done for a long time). ideas like crossvalidation and leave-one-out statistics (for example) have been around for ages. Statisticians tend not to split just once, though, unless that's unavoidable. It may depend on which statisticians you talk to $\endgroup$ – Glen_b Jul 2 '17 at 23:47
4
$\begingroup$

Not all statistical procedures split in to training/testing data, also called "cross-validation" (although the entire procedure involves a little more than that).

Rather, this is a technique that specifically is used to estimate out-of-sample error; i.e. how well will your model predict new outcomes using a new dataset? This becomes a very important issue when you have, for example, a very large number of predictors relative to the number of samples in your dataset. In such cases, it is really easy to build a model with great in-sample error but terrible out of sample error (called "over fitting"). In the cases where you have both a large number of predictors and a large number of samples, cross-validation is a necessary tool to help assess how well the model will behave when predicting on new data. It's also an important tool when choosing between competing predictive models.

On another note, cross-validation is almost always just used when trying to build a predictive model. In general, it is not very helpful for models when you are trying to estimate the effect of some treatment. For example, if you are comparing the distribution of tensile strength between materials A and B ("treatment" being material type), cross validation will not be necessary; while we do hope that our estimate of treatment effect generalizes out of sample, for most problems classic statistical theory can answer this (i.e. "standard errors" of estimates) more precisely than cross-validation. Unfortunately, classical statistical methodology1 for standard errors doesn't hold up in the case of overfitting. Cross-validation often does much better in that case.

On the other hand, if you are trying to predict when a material will break based on 10,000 measured variables that you throw into some machine learning model based on 100,000 observations, you'll have a lot of trouble building a great model without cross validation!

I'm guessing in a lot of the physics experiments done, you are generally interested in estimation of effects. In those cases, there is very little need for cross-validation.

1One could argue that Bayesian methods with informative priors are a classical statistical methodology that addresses overfitting. But that's another discussion.

Side note: while cross-validation first appeared in the statistics literature, and is definitely used by people who call themselves statisticians, it's become a fundamental required tool in the machine learning community. Lots of stats models will work well without the use of cross-validation, but almost all models that are considered "machine learning predictive models" need cross-validation, as they often require selection of tuning parameters, which is almost impossible to do without cross-validation.

$\endgroup$
  • $\begingroup$ I think that your latest edit about the prediction/inference distinction is a bit off and prone to misinterpretation (which I might do right now). If anything, if we infer that material A is stronger than B we want this to hold out-of-sample too. Plus, such an idea would ignore bootstrap/permutations approaches. And the example is a bit off; an $n \approx p$ is not really saved by a train-test-split approach but rather from regularisation. $\endgroup$ – usεr11852 Jul 1 '17 at 23:57
  • $\begingroup$ @usεr11852: yes, but it's nearly impossible to pick reasonable regularization penalties without cross-validation (other than thinking about penalties as Bayesian priors, but that's hard with black box models!). And while we do want our results in comparing A to B to hold out of sample, this typically is not a problem that requires model tuning (like prediction often does), and with the relatively low numbers of parameters, classical statistical theory can handle this without using cross validation. $\endgroup$ – Cliff AB Jul 2 '17 at 0:52
  • $\begingroup$ This is a circular argument, regularisation uses cross-validation but cross-validation is done for regularisation. That's why I commented somewhat against it to begin with. I think statistical inference/causality moves away from this non-model tuning approach (see for example 2016 Johansson et al. "Learning representations for counterfactual inference" - such a messy beautiful paper). Finally Fundamental Physics research when presented it hard problems can also rely on ML (eg. the Higgs Boson Machine Learning Challenge) approaches. $\endgroup$ – usεr11852 Jul 2 '17 at 12:19
  • $\begingroup$ @usεr11852 Regularization does not "use" cross-validation, but rather your tuning parameter for regularization is chosen using cross validation. For example, see glment's cv.glmnet for the whole procedure in a nice compact function. $\endgroup$ – Cliff AB Jul 2 '17 at 16:52
  • 1
    $\begingroup$ Also, I never made the claim that physics research cannot use ML approaches nor cross-validation! I was only explaining that cross-validation is typically used specifically for choosing between complex models/tuning parameters in predictive models, and that in many classic physics experiments, cross-validation is not necessary. So what physicists do with that data is not necessarily at odds with what statisticians would do with that data, which I believe was the core of the OP's question. $\endgroup$ – Cliff AB Jul 2 '17 at 16:56
3
$\begingroup$

Being (analytical) chemist, I encounter both approaches: analytical calculation of figures of merit [mostly for univariate regression] as well as direct measurement of predictive figures of merit.
The train/test splitting to me is the "little brother" of a validation experiment to measure prediction quality.


Long answer:

The typical experiments we do e.g. in undergraduate physical chemistry use univariate regression. The property of interest are often the model parameters, e.g. the time constant when measuring reaction kinetics, but sometimes also predictions (e.g. univariate linear calibration to predict/measure some value of interest).
These situations are very benign in terms of not overfitting: there's usually a comfortable number of degrees of freedom left after all parameters are estimated, and they are used to train (as in education) students with classical confidence or prediction interval calculation, and classical error propagation - they were developed for these situations. And even if the situation is not entirely textbook-like (e.g. I have structure in my data, e.g. in the kinetics I'd expect the data is better described by variance between runs of the reaction + variance between measurements in a run than by a plain one-variance-only approach), I can typically have enough runs of the experiment to still get useful results.

However, in my professional life, I deal with spectroscopic data sets (typically 100s to 1000s of variates $p$) and moreover with rather limited sets of independent cases (samples) $n$. Often $n < p$, so we use regularization of which it is not always easy to say how many degrees of freedom we use, and in addition we try to at least somewhat compensate for the small $n$ by using (large) numbers of almost repeated measurements - which leaves us with an unknown effective $n$. Without knowing $n$ or $df$, the classical approaches don't work. But as I'm mostly doing predictions, I always have a very direct possibility of measuring the predictive ability of my model: I do predictions, and compare them to reference values.

This approach is actually very powerful (though costly due to increased experimental effort), as it allows me to probe predictive quality also for conditions that were not covered in the training/calibration data. E.g. I can measure how predictive quality deteriorates with extrapolation (extrapolation includes also e.g. measurements made, say, a month after the training data was acquired), I can probe the ruggedness against confounding factors that I expect to be important, etc. In other words, we can study the behaviour of our model just as we study the behavior of any other system: we probe certain points, or perturb it and look at the change in the system's answer, etc.

I'd say that the more important predictive quality is (and the higher the risk of overfitting) the more we tend to prefer direct measurements of predictive quality rather than analytically derived numbers. (Of course we could have included all those confounders also into the design of the training experiment). Some areas such as medical diagnostics demand that proper validation studies are performed before the model is "let loose" on real patients.

The train/test splitting (whether hold out* or cross validation or out-of-bootstrap or ...) takes this one step easier. We save the extra experiment and do not extrapolate (we only generalize to predicting unknown independent cases of the very same distribution of the training data). I'd describe this as a verification rather than validation (although validation is deeply in the terminology here). This is often the pragmatic way to go if there are not too high demands on the precision of the figures of merit (they may not need to be known very precisely in a proof-of-concept scenario).

* do not confuse a single random split into train and test with a properly designed study to measure prediction quality.

$\endgroup$
  • 2
    $\begingroup$ +1 for pointing out the difference in verification and validation. $\endgroup$ – forecaster Jul 2 '17 at 22:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.