I have a question about calculation and reporting of statistics combining data from replication studies. Our techniques don't involve ML, but ML techniques have me thinking about this.
Often in ML there is a hold out validation set. Performance of a model on this set gives an unbiased estimate of out-of-sample performance. Often times, after a model is selected and performance estimated, it is retrained using all data (combining the training, test, and validation sets) for production / usage. The performance of the retrained model does not provide an unbiased estimate of out-of-sample performance.
We do case/control studies that do not involve ML models.
- We divide our data into a discovery and a (smaller) validation set.
- We find candidate solutions on the discovery set and test their significance on the discovery set using an exact test.
- For those that are significant we then take random samples of the discovery set with the same size as the validation set and test group association, if 95% of random samples have the same association as on the discovery set as a whole we keep those solutions (this is a test against noise but mainly used to reduce our candidate solution pool prior to validation).
- We then calculate significance on the validation test and apply MT correction using the Holm-Bonferroni method to determine final significance level.
- For those that replicate, we calculate various other statistics of interest such as odds ratio.
- We report the discovery and validation results.
It seems to me that doing a final test combining the discovery and validation sets and calculating statistics would give a better estimate of the population since the combined set is larger. My thesis advisor disagrees (and is probably right!) since the discovery set would bias this final test. My thinking: unlike ML, we aren't testing performance on a set we trained on, we are simply calculating statistics for a solution (I suppose this could be argued to be the same thing). We already know that the results are significant and replicate on the validation set, so there's no bias in what we call significant. Furthermore if, somehow, a test was significant on both discovery and validation, but not on the combined set (extremely rare) it could be discarded, improving confidence in the solution.
Would it be appropriate to calculate and report statistics (like odds ratio) on the full data?
Would it be appropriate to calculate significance (whether odds ratio p-value or exact test p-value) on the full data? If so would it need to be corrected?