Many readers familiar with scientific articles in the field of deep learning-based computer vision might have observed a common practice: the absence of statistical significance tests for comparing the algorithms. Whether the focus is on classification, semantic segmentation, object detection, or other tasks, research papers typically present comparison tables showcasing state-of-the-art approaches versus their own, using various metrics like accuracy, IoU, F1-score, and more. Yet, the application of statistical tests to demonstrate the superiority of one method over another is conspicuously lacking. This raises several questions:
- Why is the use of statistical tests infrequent in these articles?
- What's the rationale behind reporting performance based solely on the results from the final training epoch?
- How can we be certain that the observed differences, sometimes of only a few percentage points, are statistically significant?
- Is the absence of statistical tests due to the time and cost associated with repeated model training?
- Is it assumed that having a substantial amount of data makes test performance reliable? If so, how much data is considered sufficient to trust the metrics?
- Can we estimate confidence intervals by examining the variability of an accuracy metric over time within a single training run, or is it necessary to rely on multiple training runs to obtain this information?