In ML we're indoctrinated about the train/validation/test split. A simple split will provide a point estimate of the error for validation and one for test. If a k-fold approach is used we might get a couple more points and a rudimentary sense of mean and variance of the error distribution. While the practical aspects of the approach seem clear (many modern models can take hours/days to train and we cannot afford to run a 50-fold cross validation), I still fail to see how many ML papers (or, for that matter, applied business applications) can get away with claiming decimal level improvements on a single error point estimate from a hold out. Wouldn't it make sense to at least run some sort of bootstrap to assess the error distribution to be able to claim "our new model outperforms SOTA with 75% likelihood"? I mean, that doesn't sound as cool as "our model outperforms SOTA", but wouldn't the former be the right expectation for the scientific community?
-
$\begingroup$ I am trying to understand the issue/question. Let me repeat it in my words to see whether I get it. So the problem is that models are trained only once (because it is costly), and the models are only compared in a single cross validation step. Based on that validation one can make statements like "outperforms with 75%". The point of the question is that such statement only relates to the validation of a single particular training. It does not express how models vary for different training sets and whether with a different training set other models or hyperparameters could have been better. $\endgroup$– Sextus EmpiricusOct 29, 2021 at 20:02
-
1$\begingroup$ The main point is that a single estimate of the error provides no information about the error distribution and its variability. So, when I say I split in 80% train and 20% validation, and x this is my validation error, I provide no context about where x falls in the distribution of all x's. My one x could be an inlier, outlier or spot on the mean, but I'd have no idea about that. $\endgroup$– KenOct 29, 2021 at 20:14
-
$\begingroup$ On the other side, crossvalidation and/or bootstrapping can provide a sense for the error distribution. This is similar to stats where you can estimate a parameter, associate confidence intervals, and have derive conclusions based on things like p-values. Imagine you have a baseline model with ROC=0.87. Your new model has a a point estimate of the validation error of ROC=0.89. Seems like a better model. But what if the first model has a standard deviation of 0.01, and the second has one of 0.05. Which one would you pick? $\endgroup$– KenOct 29, 2021 at 20:14
-
$\begingroup$ But it revolves around this ambiguity right? On the one hand, when we have a single training done them we can use cross-validation to test which model makes the best predictions and wee can express that with a sentence like "outperforms with 75%". But on the other hand, that expression "outperforms with 75%" does not tell the entire story. The model only outperforms by that score for a single particular training. With other training sets it might perform less good. $\endgroup$– Sextus EmpiricusOct 29, 2021 at 20:29
-
$\begingroup$ I think we diverge when you say Based on that validation one can make statements like "outperforms with 75%".. What I'm saying is that "based on that validation one cannot make statements like "outperforms with 75% confidence". $\endgroup$– KenOct 29, 2021 at 20:48
1 Answer
The current trend in machine learning research is to train huge models. Let me quote one article
At OpenAI, an important machine-learning think tank, researchers recently designed and trained a much-lauded deep-learning language system called GPT-3 at the cost of more than $4 million. Even though they made a mistake when they implemented the system, they didn't fix it, explaining simply in a supplement to their scholarly publication that "due to the cost of training, it wasn't feasible to retrain the model."
If a model is to expensive to train to fix a bug, certainly nobody is going to bother with $k$-fold cross-validation.
Moreover, if you have a huge dataset and subsample it, the subsample would be still huge enough to drive the variance low. Of course assuming that re-training would be deterministic, what is not true as there are known examples of non-reproducible results in machine learning, or getting different results with different random seeds.
-
2$\begingroup$ It seems to be the key that because of the large amount of data the variance is gonna be low so people do not care much to determine it with a k-fold approach. And, anyway, the error/variation that the randomness of the training data is gonna bring is not only due to the variations in sampling, but also due to bias in sampling (for instance, if a selfdriving car is gonna mistake the moon for a traffic light, then this is not an error/deviation that a k-fold approach would have predicted) $\endgroup$ Oct 29, 2021 at 20:18