Differentiating between "exploratory" vs "confirmative" p-values

Question

In the paper Gaus et. al. (2015), two key principles are outlined for interpreting p-values. My question is about the second principle: is it common to make a distinction between exploratory and confirmatory p-values? I haven't found other examples in the literature.

Here is the statement of the principle from the paper:

The second principle is to differentiate between exploratory and confirmative p-values. An exploratory interpretation of a significant p-value typically establishes a new hypothesis. By contrast, a confirmative interpretation of a significant p-value can be considered as “statistical proof” for a hypothesis established previously.

These sections from later in the paper help to clarify:

In experimental research, one has to strictly distinguish between two steps: hypothesis generation and hypothesis confirmation. It is very important that the data for the hypothesis confirmation is independent from the data used for the generation of the hypothesis.

In principal, an exploratory analysis can only generate hypotheses, but it can never prove a hypothesis. If an exploratory significance has been obtained, then a hypothesis is generated. However, descriptive statistical results and other considerations should be used to decide, whether further research in order to confirm this hypothesis is worthwhile.

Furthermore, one needs to remember the saying that a precise answer (confirmative testing) is only possible for a precise question (hypothesis)

Answer 1

Interesting article.

is it common to make a distinction between exploratory and confirmatory p-values?

My short answer is that it is uncommon.

In the abstract authors said that

“No other statistical result is misinterpreted as often as p-value”

I agree. However It seems me that, unfortunately, this article do not help too much.

I'm not familiar with Pharmacology literature but the arguments sound general. The distinction between Exploratory Versus Confirmatory p-value, what the article is focused on, sound unusual to me. This can be my fault but I never see it in statistics/econometrics books and in Wiky this distinction is not mentioned: https://en.wikipedia.org/wiki/P-value

Explanatory p-value is presented as something useful for “hypothesis generation from data”. This sound strange to me. Indeed it seems me that the main source for “hypothesis generation” is theory or any kind of information available at priori (from previous research for example) and in any case before to see the data. It seems me that the correct interpretation of p-value is something like what authors named “confirmative”. p-value is for hypothesis testing not “hypothesis generation”.

Moreover p-value summarize classical hypothesis testing results. If the above distinction make sense it is so also for: t-stat F-stat, LR ratio, ecc. It is not common.

Indeed the paragraph: “Confirmative testing needs a hypothesis and a level of significance both established a priori” [pag 3], sound good for me. However other statements like the follow sound me dubious:

In experimental research, one has to strictly distinguish between two steps: hypothesis generation and hypothesis confirmation. It is very important that the data for the hypothesis confirmation is independent from the data used for the generation of the hypothesis. [pag 3]

Actually If the two set of data are independent problems like $p$-hacking go away. However in such situation (independent sets) as is possible that the first set give us reliable information for the second? If it not, why “hypothesis generation” come from first sample and work for the second?

Moreover, statements like following put more doubt yet:

Not all of the significant results are clearly exploratory or undoubtedly confirmative. Some are somewhere in between exploratory and confirmative. We propose to consider exploratory and confirmative as two pole positions with a continuum in between. ... . However, such hybrid significance testing should be avoided as far as possible. [pag 6]

Finally this point seems me important:

A hypothesis can be gained by intuition or through theoretical considerations, but mostly it is generated by an exploratory analysis of data. [pag 3]

In my view the point is that theory can be data-driven too. However this do not mean that related/previous p-value change his meaning, is the specific result that change his role. It is a finding in first research and become a guess in the following.