11
$\begingroup$

I read various stats/biostats textbooks, including Casella and Lehmann's book chapter on regression. Most of time, the textbook will report a p-value for significance of the parameters after regressing against some model. Then there is a model selection procedure followed afterwards.

However, those books will never touch upon cross validation (CV) or talk about using a test/training split. I learned CV and Monte Carlo cross-validation (MCCV) from machine learning books and rarely have I seen stat books covering CV.

Why were we not taught cross validation in stats? Or is it not practiced by statisticians in general? Or somehow that model selection procedure becomes superior to using testing data for model selection? Does a biostatistician/practicing statistician use CV in model selection in general?

$\endgroup$
7
  • 12
    $\begingroup$ What's a "regular" textbook? Cross-validation is addressed in standard texts such as Elements of Statistical Learning. $\endgroup$
    – Sycorax
    Sep 27 at 22:22
  • 2
    $\begingroup$ Like many things in statistics, it can take a long time to go from statistics papers to what is in the "standard" textbooks (I was first learning about cross validation in the late 80s to mid 90s but it had been around for a long time in statistics by then already -- e.g. Stone was writing in the 70s, as was Allen). $\endgroup$
    – Glen_b
    Sep 27 at 23:49
  • 1
    $\begingroup$ I believe that cross-validation does appear in Tukey's "Exploratory Data Analysis", which at one time was a standard text as well. $\endgroup$
    – Flounderer
    Sep 28 at 6:55
  • 2
    $\begingroup$ @Flounderer You're thinking of the sequel by Mosteller and Tukey. $\endgroup$
    – Nick Cox
    Sep 28 at 8:59
  • 2
17
$\begingroup$

I can't say with 100% certainty, but I can give you my two cents.

Let's start with the difference in philosophy between statistics (as practised in the books mentioned) and machine learning. The former is (usually, but not always) concerned with some sort of inference. There is usually a latent parameter, like the sample mean or the effect of a novel drug, which requires estimation as well as a statement on the precision of the estimate. The latter (usually, but not always) eschews estimating anything except the conditional mean (be it a regression or a probability in the case of some classification models).

Thus "model selection" in each context means something slightly different by virtue of having different goals. Cross validation is a means of selecting models by means of estimating their generalization error. It is, therefore, primarily a tool for predictive modelling. But statistics (again usually, but always) is not concerned with generalization error as much as it is concerned with parsimony (for example), hence statistics don't use CV to select models.

Prediction from a statisticians point of view is not absent of cross validation. Indeed, Frank Harrell's Regression Modelling Strategies mentions the technique, but that book is primarily concerned with the development of predictions models for use in the clinic.

$\endgroup$
5
  • $\begingroup$ I think if we want to make inference for some predictive outcome via modeling, the CV does not provide CI in general. One probably could do bootstraps. However, that needs some non-trivial assumption on sample distribution. Even if statistician uses CV, do they care for CI in CV in this case? $\endgroup$
    – user45765
    Sep 27 at 22:29
  • $\begingroup$ @user45765 Cross-validation is used in semi/nonparametric estimation and inference (e.g. one-step estimators - Bickel et al. (1993) and TMLE - van der Laan, Rose (2012)). These nonparametric methods allow for the use of adaptive model selection methods like cross-validation within the estimation procedure while still allowing for valid and efficient inference. Cross-validation (or more generally Ensemble/SuperLearning) is a key part of the TMLE methodology. $\endgroup$
    – Lars
    Sep 28 at 3:59
  • 3
    $\begingroup$ Regression Modeling Strategies is intended for prediction models in general. And resampling is not just used for model selection (in fact model selection is usually not even a good idea) but is used for estimating the performance of a pre-specified model in an unbiased fashion. $\endgroup$ Sep 28 at 11:42
  • $\begingroup$ @FrankHarrell there’s contradiction in your answer. Don’t we select model based on their performance? So we end up using CV for model selection and it does weaken the value of this technique. $\endgroup$
    – Aksakal
    Oct 8 at 13:42
  • $\begingroup$ "Model selection" is overused, as opposed to model specification. Most of the time we should assess performance of the prespecified model and not waste some of the information in the data on feature selection. $\endgroup$ Oct 8 at 14:52
8
$\begingroup$

The main reason is that almost all book authors are in inference statistics. In particular, bio statistics is heavy on this aspect. A lot of stats used in regulated industries, such as banking is guilty of this too. Question like "what caused this? Did this cause that?" are usually asked from the inference point of view.

Cross-validation is of interest to forecasters. If you run into a book written by people who are in predictive field, then you should see the discussion of cross-validation. A good example is Hyndman's book, see here. I personally look at p-values mostly when asked to, in our industry we have governance folks who love this type of nonsense and request us to show a ton of pointless metrics.

$\endgroup$
3
$\begingroup$

Probability

If you consider a really strict delineation of probability and statistics, the former is about mathematically describing how likely it is for an event to occur, or a proposition to be true. You can have a textbook or a course that is about probability, without entering the field of statistics at all.

Classical examples include drawing different colored balls out of an urn, combinations in a lottery, or drawing cards from a deck.

Statistics

Statistics, then, is about describing either probability distributions, populations or samples drawn from a population. Parameters that can be used to describe those are, for example, mean and standard deviation. In this sense, statistics is about describing the results of observations of random variables, or any sample or population that is not necessarily random.

A textbook that takes this view of statistics would include the definitions for those terms, and then various estimators that can be used to get at the parameters that might have produced a certain sample (given a probability distribution, or a random process), and how to judge the correctness of those estimates.

Now, it is entirely plausible that a textbook would stay entirely within this definition of statistics: Describing populations or samples, and using probability distributions to make inference on how like it is that we saw a certain sample -- without entering the world of statistical modeling, where cross validation belongs.

Why not cross validation?

Some textbooks, even holding the view described above, might still include linear regression: its parameters can still be considered estimates that can be calculated from a sample. It can be, of course, used as a predictive model, and thus subjected to cross validation -- but once you start using cross validation to make judgements about what terms to include in your model, you step away from the strict definition of the parameters of the linear model being estimates of the population, calculated from a sample drawn from it.

Thus you could say that cross validation is already venturing in to the field of applied statistics.

$\endgroup$
4
  • 4
    $\begingroup$ Don't use cross-validation to decide which terms to include unless you are prepared to be depressed by how poorly it does this (i.e., great volatility in which terms are selected). $\endgroup$ Sep 28 at 11:43
  • $\begingroup$ @FrankHarrell: agree, but OTOH, I find resampling a very intuitive way to visualize how unstable/volatile data-driven model optimization is. I wouldn't say resampling does particularly poorly in that respect, but that data-driven model optimization is very data-hungry, and in practice often combined with particularly unsuitable target figures of merit. $\endgroup$ Sep 29 at 7:39
  • $\begingroup$ The point I tried to make is that once you start tuning your model, you step away from the view that the coefficients are just estimates for some measures that summarize your sample, similar to mean or standard deviation. Feel free to edit the answer if the point can be made clearer or exclude any suggesting that it should be used for it :) $\endgroup$ Sep 29 at 8:50
  • 1
    $\begingroup$ @cbeleitesunhappywithSX The bootstrap is superb at depicting the instability of "selected" features. What I was getting at is that resampling teaches us not to do feature selection in many cases, because the set of "selected" features is not reproducible. $\endgroup$ Sep 29 at 12:06
0
$\begingroup$

Descriptive vs. predictive

In, say biology textbooks, the focus is on describing the relevant data; the sample mean was…, the p-value for this result is… etc.

In machine learning texts, the focus is on producing models that can generalize beyond their training set; thus techniques like cross-validation are required in order to get a handle on that feature.

This oversimplifies a bit, but I think captures the essence of the difference in approach that some works take relative to others. These differences are a reflection of the different priorities and goals for these different domains that apply statistics.

Consider the Theory of Point Estimation by Lehmann and Casella 1998. They start chapter 1 with the statement "Statistics is concerned with the collection of data and with their analysis and interpretation.", which reads to me that these authors have chosen to focus more on applying statistics for analysis and interpretation rather than on applying statistics in a manner that can be reliably generalized to other cases. Similarly, they describe the result of "classical inference and decision theory" as " Such a statement about [the estimated values of the model parameters] can be viewed as a summary of the information provided by the data and may be used as a guide to action." -- again the focus is more about how to describe the data (any maybe as a basis, with some interpretation, on how to guide actions), and less about the nature of the problems involved in ML.

Finally, let me emphasize: this is a matter of what particular authors, problem domains etc. have chosen to focus on in different areas.

$\endgroup$
4
  • 4
    $\begingroup$ That is way beyond oversimplified. Statistics is concerned with estimation, hypothesis testing, decision making, and prediction. Statistical models have been used for prediction for over a century. $\endgroup$ Sep 28 at 11:43
  • 4
    $\begingroup$ I dislike how seemingly common the attitude "statistics did not understand causality/prediction/nonlinearity/etc until ML came around" has become. $\endgroup$ Sep 28 at 15:57
  • 1
    $\begingroup$ If you calculate a p-value (confidence interval, Bayes factor, credible interval...), you have gone beyond descriptive statistics. $\endgroup$
    – Dave
    Sep 28 at 17:44
  • $\begingroup$ @Dave there is the technical definition of descriptive statistics and then there's the general-usage meaning of descriptive. I'm using the latter. The idea is that in some domains the emphasis is on the more descriptive side of things and in other domains, predictive aspects dominate. A p-value describes how likely a given ensemble of data are under the null hypothesis; a p-value doesn't give you a good way to predict how future data will behave. $\endgroup$
    – Dave
    Sep 28 at 18:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.