I feel that your statement about study design misrepresents what it is about.
Traditionally, sample size determination is done as part of the design phase. To this end, one has to have an understanding of the baseline performance, such as the mean and standard deviation of the metric in question. One might use recent historical data to obtain reasonable estimates.
Study Design Basics
Study design is all about risk management and should not be based on empirical data such as mean and standard deviation. Rather study design should be based on what is a minimally useful normalised effect size (inter-group difference over intra-group variation) balanced against maximum acceptable risk. If the effect size is useful you want to ensure your experiment detects it at a level of risk that is acceptable to stakeholders. To see why look at a basic sample size calculator:
n=2(Zα+Zβ)2σ2Δ2
Basically we fix a constant by deciding alpha and beta based on stakeholder sensitivity to rejecting null when it is true vs accepting it when it is false. If we use empirical data to fill in σ and Δ, then our calculation will use the same information that we would use to calculate p values, e.g. the t-test calculates
t=Δσ
Which is then transformed into Z scores. You'll notice that t has the same variables, but inverted. I.e. using empirical data in a sample size calculator will simply tell us what we would already know by looking at the p value, albeit in a transformed way.
Proposed approach
The process described in the OP is performing exploratory data analysis and appears to be attempting to use it as confirmatory analysis. To manage risks it seeks to use statistical methods to provide respectability to the output.
One of the critical needs for confirmatory analysis is the concept of independence. Hypotheses need to be tested on data that is independent of the data used to originally generate them, otherwise you end up with familywise errors as the errors at later stages will be implicitly tied to the errors at the early stage since the variation in the early samples persists and influences all subsequent analyses.
The process in the OP cannot be used to provide falsification of the hypothesis that the experiment is itself generating. If you look at the basic formula for sample size calculation above this becomes clear.
Apart from the fact that one might launch an experiment that is doomed to fail, are there any other problems with the above logic? Of course, we are not assuming a clinical setting but rather a service of some kind.
Launching an experiment doomed to fail is probably the most insignificant possible problem as all that is wasted is the resource put into the experiment. A much bigger risk is that the experiment leads to an inappropriately weighted decision that misunderstood the magnitude of risk and led to an overconfident response. Here the effect goes way beyond the resource associated with performing the experiment. A couple of examples of what it may lead to committal of:
- new investment in something flawed that will fail
- cancelling a product that could have made a positive impact
- releasing a product or service that has a negative impact on users
- fraud, if the risks were willfully not clearly detailed
Many proof of concepts start out life using informal or semi-formal processes similar to what is described in the OP. However, it must be clear to all stakeholders that it is exploratory in nature and unconfirmed by independent testing. It would be advisable to undertake a statistical design of the process to be sure you understand the risks, but to be honest this sounds like a much more difficult job than a standard study design approach based on first understanding stakeholder needs, risks and benefits. The amount of effort put into this should be proportional to the expected risk/benefits of the final use case of the research outputs. Changing the colour of a web site button probably warrants 2 minutes within a design meeting to establish these, whereas a medical device will require domain experts and a full benefit-risk analysis.
When presenting output of the analysis it must be qualified and the appropriate caveats must be attached to all downstream decisions and recommendations. All stakeholders (anyone affected by the decisions in any way) should be made fully aware of the risk associated with the data.
Conclusion
It would be difficult to see how to justify the approach without a much more robust defence and a mathematical/statistical proof of how it performs. This would be much more work than it is proposing to save.