Post-hoc hypothesis testing... would this variation work? Why or why not, exactly?

Question

Let's say I gather a ton of data in an experiment and then notice an interesting pattern unrelated to what I was testing.

So we know well enough that I can't suddenly form that pattern as the hypothesis and then use that data as evidence to test that hypothesis and get a p-value—I will need to run a new experiment.

So far so good? Okay, now consider the following approach:

Once you notice the pattern, don't tell anyone anything about it (not even subtle hints).
Just go up to one colleague who isn't likely to guess what you might have observed and ask "Is there any hypothesis you'd like to test on a dataset of this sort?" and see what they say:

• If they suggested a different hypothesis, then your "freebie" is gone and you can't go ask someone else (otherwise you'll eventually find someone who suggests the same hypothesis). Tough luck, but that's how it works.

• But if they happen to suggest the same hypothesis as yours the first time, then why not just run the test on the existing data you gathered? As long as you don't let them glean any information about the dataset from your question, it should be exactly the same as gathering new data from scratch, except that it saves all the costs, right?

Is this approach statistically valid? On the one hand it's probably making some of you cringe (me too) that we're depending on the likelihood of a human to guess the "correct" hypothesis (we're essentially turning things around and testing the experimenter!), but on the other hand from a mathematical standpoint I don't see why it shouldn't work...

Answer 1

Considering your proposal as a pure thought experiment I would say it is valid. You simply test for 2 hypothesis that are both not conditioned by the data ; whether there are formulated before or after data collection is actually not relevant as long as their formulations have nothing to do with the data.

In my opinion the point in your question is that this scenario is the expression of something important : carefully stating hypotheses. In theory; nothing stop us from discussing hypothesis with others and to prepare a hierarchy of tests based on their expected relevance/importance. Moreover, by planning it before data acquisition, we can adjust our experimental setting to these particular hypotheses.

To sum-up, your proposal may simply be a difficult-to-justify way of doing what can be done better and properly by taking more time to elaborate hypotheses (please do not misunderstand me, I know this is a complex and painstaking task that is easy to fail).