Statistical uncertainties in Deep Learning

Question

Recently I got very interested in NLP applications of deep learning. Diving into literature (on arXiv for instance) I noticed that is very unpopular to quote and estimate uncertainties on scores of ML tasks. In the era of pretrained language model (i.e. bert, gpt etc.) all further improvements quoted in papers seems to be compatible among each other within 1 or less standard deviations, making all the results statistically compatible with a fluctuation due to stochastic optimization in neural network training procedure (at fixed data-set). I am a physicist, and this looks really confusing to me when compared to the statistical treatment of experimental data performed by routine in laboratories. I am sure this question has already been discussed in the past in ML/Data Science community, could you point me some review or paper addressing this issue? Also, could you please share with me your thoughts about?

Supposing a setup with same datasets, just different model structure where the stochasticity is purely given by intrinsic randomness of the optimization procedure (SGD). What I am asking is:

1. Why uncertainties are usually not quoted in association to ml scores?
2. If uncertainties are not quoted how it is possible to compare different approaches and claim a possible improvement without a statistical confidence on the claim?

Let me propose a trivial example: I train model A on some data, and on a test set I get an f1 score of 80.0+-2.0, where I am quoting central value as the mean over N trainings and 2.0 is the standard deviation (assuming N is large enough). Then I train model B which is similar to model A but with a different topology (same dof as model A) and measure an f1 = 82.0+-(5.0). Would you claim model B is better than model A? Or would you consider the two scores to be statistically indistinguishable since they are compatible between each other in less then 1 sigma?

Answer 1

how it is possible to compare different approaches and claim a possible improvement without a statistical confidence on the claim

I think that many papers overclaim. In the subfield of giant pre-trained models evaluated on GLUE: even if they are tested on the same datasets, they are usually trained on different data so it is not possible to claim that they are "better overall". A more realistic claim is that the data and the models yield better results, with all the caveats (maybe other methods trained longer or better optimised would be better, or other methods with the same data, or the improvements on the benchmarks do not reflect real progress as was shown on NLI recently, etc.).

ML and NLP researchers and reviewers were more concerned about statistical significance years ago, see for example Dietterich (1998), a popular paper on the topic. The standard have dropped on that front for possibly several reasons:

• People realized that statistical testing and the whole p-value approach can be more harmful overall, see for example this wikipedia page on misuse of p-values and Andrew Gelman's piece. That might justify dropping hypothesis testing but not ignoring variance.
• Datasets have grown a lot since the 90s/00s. The large increase in train and test data have reduced the variance of the results quite a lot and the influence of parameter initialisation and randomness in the optimisation procedure is less important. That might justify ignoring variance.
• New researchers are less exposed to statistics as ML research distinguished itself from stats (see Breiman's "The two cultures" for example). I've noticed this personally, as a PhD student in a big "AI" public lab.

While these reasons make sense, I would say the trend went way too far. You are not the only one to be concerned. Here are a few interesting papers to realize the extent of the problem or proposing concrete solutions:

• It is hard to say that an algorithm is better than another in general, so weaker claims are about algorithms being better on specific datasets. However, even such weak claims sometimes do not hold to scrutiny! Gorman and Bedrick (2019) showed in a replication study that these results sometimes only hold when the "standard" train/test splits are used!
• In general, one cannot simply reuse the numbers of another paper directly and needs to replicate the results. But a common problem is unfair comparison due to uneven optimisation of the hyperparameters. Dodge & al. (2019) proposed to make more robust comparisons by taking into account the amount of computation used.
• Dror & al. 2017 focused on how to make broad claims based on the results on several datasets.

The problem is not specific to NLP. For example, recent work by Musgrave & al claim that recent "improvements" in the subfield of metric learning have been "at best marginal" (Figure 4 is extremely telling and concerning).

References:

• Dietterich (1998): Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms
• Gelman: The problems with p-values are not just with p-values
• Gorman & Bedrick (2019): We Need to Talk about Standard Splits
• Dodge & al. (2019): Show Your Work: Improved Reporting of Experimental Results
• Dror & al. (2017): Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets
• Musgrave & al. (2020): A Metric Learning Reality Check

Answer 2

First problem is that fitting the model once and comparing the metrics is not the best approach as discussed in the answer by EdM to the How to find a statistical significance difference of classification results? question (see the whole answer, as it seems to answer your question to great extent):

Second, instead of simply fitting the model one time to your data, you need to see how well the modeling process works in repeated application to your data set. One way to proceed would be to work with multiple bootstrap samples, say a few hundred to a thousand, of the data. For each bootstrap sample as a training set, build KNN models with each of your distance metrics, then evaluate their performances on the entire original data set as the test set. The distribution of Brier scores for each type of model over the few hundred to a thousand bootstraps could then indicate significant differences, among the models based on different distance metrics, in terms of that proper scoring rule.

With simple models, we can make some assumptions and derive the errors, for more complicated cases, as noted in the answer we can use procedures like bootstrap. Now, with deep learning models using bootstrap is problematic, because they need great computational power and time to train, where cost of the biggest models in this field is comparable to cost of a car (per single training). This is one of the reasons why there is ongoing research on models that are aware of their uncertainties, e.g. Bayesian neural networks, and many research projects look into approximating it, e.g. with using dropout in prediction phase (see blog post by Yarin Gal, but see also the critique by Ian Osband). All those approaches are based on approximations and have their pitfalls. So the answer to your question would be that it's not that simple to get meaningful estimates for the uncertainties.