2
$\begingroup$

Let's say I gather a ton of data in an experiment and then notice an interesting pattern unrelated to what I was testing.

So we know well enough that I can't suddenly form that pattern as the hypothesis and then use that data as evidence to test that hypothesis and get a p-value—I will need to run a new experiment.

So far so good? Okay, now consider the following approach:

Once you notice the pattern, don't tell anyone anything about it (not even subtle hints).
Just go up to one colleague who isn't likely to guess what you might have observed and ask "Is there any hypothesis you'd like to test on a dataset of this sort?" and see what they say:

  • If they suggested a different hypothesis, then your "freebie" is gone and you can't go ask someone else (otherwise you'll eventually find someone who suggests the same hypothesis). Tough luck, but that's how it works.

  • But if they happen to suggest the same hypothesis as yours the first time, then why not just run the test on the existing data you gathered? As long as you don't let them glean any information about the dataset from your question, it should be exactly the same as gathering new data from scratch, except that it saves all the costs, right?

Is this approach statistically valid? On the one hand it's probably making some of you cringe (me too) that we're depending on the likelihood of a human to guess the "correct" hypothesis (we're essentially turning things around and testing the experimenter!), but on the other hand from a mathematical standpoint I don't see why it shouldn't work...

$\endgroup$
  • $\begingroup$ If you get a different pattern, you may go for testing a new hypothesis as well provided it contributes to your current research objective. If it (new hypothesis) does not have any link with your main study, it is better to skip what you notice and just indicate that it could be a matter for future research. $\endgroup$ – Subhash C. Davar Oct 11 '17 at 4:41
  • $\begingroup$ @subhashc.davar: I'm asking about what is considered statistically valid, not about academic best practices! $\endgroup$ – Mehrdad Oct 11 '17 at 4:45
  • $\begingroup$ Assuming that you have in mind the validity of a hypothesis, the approach you have contemplated can not be considered valid in terms of statistics. $\endgroup$ – Subhash C. Davar Oct 11 '17 at 6:16
  • 1
    $\begingroup$ @subhashc.davar: Could you explain why? $\endgroup$ – Mehrdad Oct 11 '17 at 6:29
  • 1
    $\begingroup$ Not sure of what you mean by "to suggest the same hypothesis as yours the first time" ? $\endgroup$ – peuhp Oct 11 '17 at 7:44
1
$\begingroup$

Considering your proposal as a pure thought experiment I would say it is valid. You simply test for 2 hypothesis that are both not conditioned by the data ; whether there are formulated before or after data collection is actually not relevant as long as their formulations have nothing to do with the data.

In my opinion the point in your question is that this scenario is the expression of something important : carefully stating hypotheses. In theory; nothing stop us from discussing hypothesis with others and to prepare a hierarchy of tests based on their expected relevance/importance. Moreover, by planning it before data acquisition, we can adjust our experimental setting to these particular hypotheses.

To sum-up, your proposal may simply be a difficult-to-justify way of doing what can be done better and properly by taking more time to elaborate hypotheses (please do not misunderstand me, I know this is a complex and painstaking task that is easy to fail).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.