In the paper Gaus et. al. (2015), two key principles are outlined for interpreting p-values. My question is about the second principle: is it common to make a distinction between exploratory and confirmatory p-values? I haven't found other examples in the literature.
Here is the statement of the principle from the paper:
The second principle is to differentiate between exploratory and confirmative p-values. An exploratory interpretation of a significant p-value typically establishes a new hypothesis. By contrast, a confirmative interpretation of a significant p-value can be considered as “statistical proof” for a hypothesis established previously.
These sections from later in the paper help to clarify:
In experimental research, one has to strictly distinguish between two steps: hypothesis generation and hypothesis confirmation. It is very important that the data for the hypothesis confirmation is independent from the data used for the generation of the hypothesis.
In principal, an exploratory analysis can only generate hypotheses, but it can never prove a hypothesis. If an exploratory significance has been obtained, then a hypothesis is generated. However, descriptive statistical results and other considerations should be used to decide, whether further research in order to confirm this hypothesis is worthwhile.
Furthermore, one needs to remember the saying that a precise answer (confirmative testing) is only possible for a precise question (hypothesis)