Can I make a statement about how much larger of a delta a metric needs to be, such that a neutral conclusion become a significant one? Apologies for the convoluted title. Couldn't figure out a more elegant 1 sentence description.
Here are the details.
I have an experiment measuring a metric e.g. conversion rate, that concluded with a confidence interval straddling 0, and a p-value larger than the classic 0.05 threshold. This points to insufficient evidence to reject the null, a scenario I'm describing as a "neutral" result.
With the information I have from this experiment, is there anything wrong with me making the following claim?

*

*I can calculate the standard error from the confidence interval e.g. CI of [-0.5, 1.3] = SE of 0.9

*Can I claim that had this experiment resulted in an observed delta of 0.9, assuming S.E remains the same, my new C.I would be [0.0, 1.8] leading to a conclusion of statistical significance?

My instincts tell me there's a big flaw in this reasoning but I can't figure out where exactly.
 A: TLDR
I don't think your approach makes any sense as the sampling should be robust. Focus on the statistical power of your result instead, which you can compute using e.g. G*Power. If the power is too low, your sample is too small to confidently reject the hypothesis. If it is high enough, the effect might simply not exist.
Long answer
I personally struggle with the notion of "this would be significant if the data was different", because I don't see the benefit of this. If your sampling method is not flawed, why should you assume significant changes from another sample? Instead, I would suggest focussing on the statistical power of your result.
Statistical power refers to the notion of type 1 and type 2 error. I'm not too familiar with the underlying theory, but you basically use your resulting sample size and effect size to compute the power, i.e. the chance of falling for a type 2 error when rejecting your hypothesis. If the power is too low, your sample size is too small to actually be able to detect the effect in the population. There's different conventions for when a power is acceptable. I know that .8 and .95 are commonly used in psychology, but the values really are subjective at the end of the day.
For me, illustrations really helped to get the point across. I'm using G*Power here, which is really handy for all kinds of power computations. If you find the concept of statistical power useful, I highly recommend you check it out. Now, on to the illustrations:
First, let's assume a fairly small effect in a correlation with a sample size of 100 and the common assumption of $\alpha = .05$. You can see that the resulting power of my experiment (bottom right of the screenshot) is only 0.25 (apologies, the "," is the German convention instead of "."). In the top-part, you see that our assumed distributions are highly overlapping and that our $\beta$ is very large, i.e. we have a high probability of running into a type-2 error when rejecting our hypothesis. Accordingly, our power $1-\beta$ is very small.

Now, let's assume the same setting but with a much larger effect. You can see that the resulting power is approximately 0.99 and that the curves hardly overlap at all. So the probability of comitting a type 2 error when rejecting our hypothesis is almost negligible.

Please note that this of course does not imply that we can say that our null hypothesis is actually true, but only strengthens the assumption that our hypothesis is not true.
Coming back to your question, the metric that you probably want to address is sample size. With a growing sample size, you can actually detect smaller effects, as the resulting distributions get narrower. To illustrate, I'm gonna reuse the previous example with a small effect size of 0.1 but a larger sample of 3000 instead of 100:

You can see that the resulting power is again very high, even though the effect is still small. This is because the CI reduces with a larger sample size, which you have probably heard before. So what I would propose in your case is to compute the statistical power of your experiment and report this. You can say something like "the given sample is too small to reliably detect the assumed effect" if the power is too low and maybe compute a required sample size if the power is actually too low. If the power is high enough, I wouldn't beat around the bush and just accept that your hypothesis cannot be accepted.
Lastly, keep in mind that having too large sample sizes actually leads to $p$ hacking, as anything will become significant when the sample is large enough. This is a separate issue though and I feel like this post is way too long as it is, so I'm not going to dive into this.
I hope this helped! If there are any questions, please feel free to ask :)
A: Rather than theorizing what if the estimated effect were twice as big, you can plan the next experiment so that its standard error is twice as small. (For the experiment that's already over, there is nothing to do other than present the results clearly and truthfully.)
To make this concrete, let's suppose that you are working with a confidence interval of the form:
$$
\begin{aligned}
\bar{x} \pm z_{\alpha}\frac{\sigma}{\sqrt{n}}
\end{aligned}
$$
where $\bar{x}$ is the estimate and

*

*$z_{\alpha}$ is the critical value, $\sigma$ is the standard deviation, $n$ is the sample size

*$\sigma/\sqrt{n}$ is the standard error;

*$z_{\alpha}\sigma/\sqrt{n}$ is the margin of error.

Here $z_{\alpha}$ is determined by the significance level $(1-\alpha)$100% and $\sigma$ is a property of the population under study. You probably want to keep $\alpha$ and $z_{\alpha}$ fixed, and it's hard to change $\sigma$.
But you do get to choose the sample size $n$. You can make the margin of error (the half-width of the confidence interval) as small as you want by increasing $n$. In your case, $\bar{x} = 0.4$ and $z_{\alpha}\sigma/\sqrt{n} = 0.9$, so the margin of error is more than 2 × the estimate; the sample wasn't large enough to estimate the effect very precisely.
And here is where planning for the next experiment comes in. If an effect size of 0.3 is the smallest effect size of practical interest, you'd want the margin of error to be less than 0.3. So you use the results from the concluded experiment to get an estimate $\hat{\sigma}$ of the standard deviation $\sigma$ and then you solve for $n$ in the formula $z_{\alpha}\hat{\sigma}/\sqrt{n} < 0.3$.
PS: I deliberately choose the "smallest valuable" effect to be smaller than the effect observed in the pilot study. This is the effect that you would regret to miss as "non-significant" because the followup experiment wasn't large enough to detect it at the $(1-\alpha)$100% level.
