Before I begin, I would like to note that I am familiar with the ordinary debate between one-tailed and two-tailed tests and the gazillions of questions already on Cross Validated about the matter.
Essentially, though, it seems that there is no theoretical reason why a sort of "pick the best tail and test it" method would not exist. In fact, this is more-or-less what happens in practice, the results are just bootstrapped out of what is essentially a larger dataset.
For example, I am interested in choosing the better of treatment A vs treatment B. The ordinary scientific process here would be to gather a small exploratory dataset, take a look at the results, and decide informally that there is sufficient evidence that one or the other actually performed better to warrant further study. I then gather a larger dataset, and perform a one-tailed test in the direction suggested by my exploratory analysis.
However, putting the small study and the large study together, it seems that in reality what I have is a single process that simultaneously picks a direction and also performs a test in that direction. This seems to contrast, in a theoretical sense, with the usual advice to not use your data to "pick a direction" to subsequently test - but this advice seems to stem strictly from the practical consideration that the usual formulae are not designed to use the same dataset to both choose a direction and also test in that direction.
Taking this into consideration, though, it seems like perhaps splitting my data into an "exploratory" subset and a "confirmatory" subset is maybe not the best usage of my limited data resources. In the sense that perhaps stronger evidence for my one-tailed hypothesis actually exists in the data than my process of splitting the data into two separate experiments would produce in terms of $p$-value.
Is my reasoning here flawed somehow? If not, does such a superior method exist for performing the "pick the best tail and test it" procedure?
