I read everywhere that the ideal way of training a model would be to e.g.: run k-fold learning for hyperparameter optimization on 80-90% of the dataset, then test the best model on the rest. As far as I understand, in each iteration of the k-fold, we receive an estimation of the generalization capabilities of our model. Now, given that the dataset is sufficiently large, this performance estimate will be the same as if the same model is run on the test set, is it not? Focusing on a single fold, we have a split of e.g.: 60-20-20 of train/validation/test sets, where the 20-20 splits are both unused for training and are from the same distribution, given a proper random sampling. So then, if we optimize our hyperparameters for the validation split of 20, it should largely be equivalent to optimizing it for the test split of 20. In the end, we want the best generalizing model, so why bother with the test set at all? Using the test set as the final evaluation of our model's performance will be the same biased estimate as was with the validation set either way. We would need a different dataset altogether from a different source to properly evaluate the true, unbiased generalization of our model, wouldn't we?

In short, why not just run hyperparameter optimization with k-folds on the entire dataset, then pick the best model and publish its k-fold averaged performance with disclaimers that it was not tested on different sets? It sounds more correct to me, than pretending that the test split from the same dataset is a better estimator.

  • $\begingroup$ Let's say you take a random sample from your population (50000 observations). You then randomly split this sample in two samples (30000 train and 20000 test). How is it different from taking two random samples (30000 and 20000) from your population, as you suggest? $\endgroup$
    – J-J-J
    Commented Apr 18, 2023 at 13:24
  • $\begingroup$ I think I rather suggest, that for the true evaluation of the generalization capability, we should test the model on random samples from a different population, on which we want to solve the same problem, but lack the amount of data necessary to train another model. For example, we train a model on medical data recorded in the US, which is abundant, then test it on EU data, which is harder to access in large quantities, or on the data of a specific country, which is significantly less. These datasets could have significant differences to which generalization would be beneficial. $\endgroup$
    – oliver.c
    Commented Apr 18, 2023 at 14:47
  • 1
    $\begingroup$ Then your real problem is not really with the test/train split method, it's rather that you're initially unable to gather a random sample of your real population of interest (in your example, your population of interest is Europe+US, not US). Now, there are criticisms of the train/test split method, and alternative validation methods exist (see fharrell.com/post/split-val for example), but if your initial sample is not a random sample from your real population of interest, you're likely to run into generalization problems anyway. $\endgroup$
    – J-J-J
    Commented Apr 18, 2023 at 14:55
  • $\begingroup$ By the way, if the cost relative to gathering a true simple random sample is prohibitive, a possible solution might be to look into complex survey designs (e.g. publichealth.columbia.edu/research/population-health-methods/…). But it really depends on the specific problem at hand, and it would be probably better to ask a specific question about it. Even if you're not in charge of the data collection process, your concern about the relevance of the sample is something you can bring to the attention of people in charge. $\endgroup$
    – J-J-J
    Commented Apr 18, 2023 at 18:20
  • 1
    $\begingroup$ Thank you for your answers, I think I understand it a bit more now. My example was semi-relevant to our work. It mostly boils down to my empirical experience, that different biological datasets can be vastly different because of different sampling/filtering/measuring methods, even though they should represent the same underlying data and population. So, often times I see models that perform well on their own test set, obviously, then completely fail on another set from a similar but different source. $\endgroup$
    – oliver.c
    Commented Apr 19, 2023 at 8:50


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.