This inspiring answer describes a variant of hypothesis, and I want to analysis its property further. Basically, it considers a two-sided test and interprets the $p$-value as a measure of how strong the evidence is that our estimate has the correct direction (i.e., positive or negative effect). It's also indicated that the $p$-value has something to do with the sample's signal-to-noise ratio, from which this question arises.
Its methodology is summarized below
- Form a null hypothesis whose alternative hypothesis is two-tailed, e.g. $H_0: \mu = 42$ v.s. $H_1: \mu \ne 42$.
- Collect some data, and merge it to the sample at hand, if any.
- Calculate the $p$-value.
- See if the $p$-value is less than a threshold, say 0.05.
- If it is, reject $H_0$, and conclude the direction as indicated by the sample;
- Go to the outermost 2 if you are willing to continue investigating;
- Otherwise, stop and declare that you can't conclude the direction without collecting more data.
From a Neyman-Pearson perspective, this is horrible as the type I error rate would be 100%. However, this is fine because equality null hypotheses on a continuous RV is almost surely false, so the error rate "conditional on it" is meaningless. Instead, this approach looks Fisherian, because the steps are decided a posterior, and it's iterative.
Now, I want to learn more about its property. It is pointed out by its author that by doing so, we avoid drawing a conclusion from a sample whose signal-to-noise ratio is too small. Intuitively it makes sense, but I'm not really familiar with the concept of "signal-to-noise ratio", so I need you to explain it in this context. For example,
- What's its definition? (Wikipedia say there are alternative definitions)
- How is it related to the $p$-value?
- How is it related to the sample size?
- What does a high S/N imply?
As a bonus, I'm curious to know the probability of concluding a wrong direction using this method, but feel free to skip this one if I'm asking too many questions.