The idea of adaptive data analysis is that you alter your plan for analyzing the data as you learn more about it. In the case of exploratory data analysis (EDA), this is generally a good idea (you are often looking for unforeseen patterns in the data), but for a confirmatory study, this is widely accepted as a very flawed method of analysis (unless all the steps are clearly defined and properly planned out in advanced).
That being said, adaptive data analysis is typically how many researchers actually conduct their analyses, much to the dismay of statisticians. As such, if one could do this in a statistical valid manner, it would revolutionize statistical practice.
The following Science article claims to have found a method for doing such (I apologize for the paywall, but if you are at a university, you likely have access): Dwork et al, 2015, The reusable holdout: Preserving validity in adaptive data analysis.
Personally, I've always been skeptical of statistics articles published in Science, and this one is no different. In fact, after reading through the article twice, including the supplementary material, I cannot understand (at all) why the authors claim that their method prevents over-fitting.
My understanding is that they have a holdout dataset, which they will reuse. They seem to claim by "fuzzing" the output of the confirmatory analysis on the holdout dataset, over-fitting will be prevented (it is worth noting that the fuzzing seems to be just adding noise if the calculated statistic on the training data is sufficiently far from the calculated statistic on the holdout data). As far as I can tell, there is no real reason this should prevent over-fitting.
Am I mistaken on what the authors are proposing? Is there some subtle effect that I'm overlooking? Or has Science endorsed the worst statistical practice to date?