Is it sufficient to report only the result of cross validation in research paper?

Question

I'm working on my master's in developing a machine-learning model to predict classes of biomedical images from a microscope. These images are

• collected separately from each patient. For example, I have ~70 cases and each one of them contains ~50 images (small dataset).
• collected with different conditions. For example, images in each case were taken at different lighting intensities.

The second point was performed intentionally to reflect the actual use in medical healthcare. In the current state, I only measured the performance of my model, i.e., accuracy and AUC, using cross-validation.

However, I have no idea about how should I split my dataset to use as a test set as the performance of the model is going to depend on which cases are selected. For example, if the selected cases in the training set are collected in a similar condition as the cases in the test set, I would surely get high accuracy which is not may not happen in real use.

Thus, is it enough to report the result from cross-validation without including the result from a hold-out test set?

A possible solution is collecting more data to use as a test set. However, that could take a long in the current state. So, I would like to seek other methods if it's possible.

Thank you for reading

Answer 1

Your concern about the potential bias introduced by the selection of cases for testing is valid, and it's good that you're thinking about ways to mitigate it. Here are some considerations and potential strategies:

1. Hold-Out Test Set: While cross-validation provides a good estimate of model performance, having a separate hold-out test set is valuable. It helps assess how well the model generalizes to new, unseen data. If collecting additional data for a test set is challenging, consider carefully selecting a representative subset from your existing dataset to act as a hold-out set.

2. Stratified Sampling: When splitting your data, use stratified sampling to ensure that each subset (training and test) maintains a similar distribution of conditions, lighting intensities, or any other relevant factors. Stratified sampling helps ensure that each class or condition is proportionally represented in both training and test sets.

3. Data Augmentation: Since you have variability in lighting conditions, consider using data augmentation techniques. This involves applying random transformations to your images (rotations, flips, brightness adjustments) to artificially increase the diversity of your dataset.

4. Model Evaluation Metrics: Besides accuracy and AUC, consider using additional evaluation metrics that are more robust to imbalanced datasets or variations in class distribution. For instance, precision, recall, F1 score, or confusion matrices can provide more insights.

5. External Validation: If possible, consider collaborating with other research groups or institutions to obtain an external dataset for validation. This adds an extra layer of validation from data collected independently.

6. Real-world Simulation: Simulate real-world conditions in your test set. For instance, if lighting conditions vary, ensure that your test set reflects this variability.

7. Robustness Testing: Test your model's robustness by introducing variations similar to those you expect in real-world scenarios. This could include different lighting conditions, image resolutions, or other factors.

8. Documentation: Clearly document your data splitting strategy, including any considerations or biases introduced. Transparent reporting helps others understand the limitations of your study.

In summary, while cross-validation is valuable, having a separate hold-out test set with careful consideration of representation across conditions can enhance the robustness of your model evaluation. It's about finding a balance given the constraints of your dataset and the real-world scenarios you aim to address.