We are working on an ML model that predicts a numeric result (call it $\hat{x}$). Eventually, we will perform an A/B test, where the metric is a function that takes $\hat{x}$ as an input (call it $f(\hat{x})$). Our control group will get the input from the old source (call it $x$, so its metric is calculated as $f(x)$), and the variant group will get the input as an output from the model (thus its metric is calculated as $f(\hat{x})$). We will then test to see whether the variant group metric's mean is higher by a specified effect size than that of the control group.
We want to perform the experiment's power analysis right away, even before the model is finished, so that we understand the experiment and know whether the required sample size is feasible and thus whether the improvement promised by the model is testable in this way. We have a reasonable effect size in mind (call it $\bar{x}$), as well as the standard $\alpha$ (0.05) and power (80%).
We can calculate $f(x)$ on past data, and it is from that data that we hope to shape the power analysis. One thing I am realizing is that the choice of past population from which to get the standard deviation $s$ with which to calculate the power analysis input (Cohen's $d = \bar{x}/s$) can make an enormous impact on what sample size is required for the future experiment. The sample size output is extremely sensitive to that $s$, in our case. So my question is, what is a good population from which to get the $s$? Some of us feel we should use the initial training data for the model. Others feel that constitutes some sort of leakage and we should use the initial test data. It may be the answer varies, depending on other factors. If the train / test choice is not clear cut, what are some guidelines to keep in mind?