What is the term for when, through accident or intent, many hypothesis tests are conducted until an 'interesting' result is found? That is, the experiment (or variations on it) are repeated until a statistically significant result is found that other researchers would probably not be able to replicate (not without themselves running the test many times).
A more serious allusion in this quote by Ronald Coase:
If you torture the data enough, nature will always confess
- This from Hadley Wickham:
You can only use an observation once to confirm a hypothesis. As soon as you use it more than once you’re back to doing exploratory analysis. This means to do hypothesis confirmation you need to “preregister” (write out in advance) your analysis plan, and not deviate from it even when you have seen the data.
What is this phenomena called - does it have a noun or a verb? That is, what is it called when someone does not adhere to statistical best practice, and repeats tests (perhaps with variations) until they find a result that stands out in some way?