Studies with small sample sizes I'm asking myself the question of why studies with small sample sizes are not as convincing as those with larger sample sizes, and when this becomes a statistical issue. A complaint I've heard a lot with studies is while they show a certain "desired" result (e.g. the treatment group had lower cholesterol than the placebo group), the sample size was small.
Statistically, if you have the sample size to detect an effect, then the sample size was large enough in that sense. So at that point, on what grounds could you claim a higher sample size would be better? To get more exact confidence intervals if they are asymptotic?
Personally I would feel more convinced with a sample of 1000 rather than 500, but I can't identify technically where this makes a difference aside from precision.
 A: Let's use reductio ad absurdum to see how the "if you have the sample size to detect an effect, then the sample size was large enough in that sense" logic doesn't hold. Imagine that you had a sample of size $N=1$. In such a sample, you can either observe a result that confirms or disagrees with your hypothesis. Unless you can make a strong hypothesis like "it should never happen" (but then, you don't need statistics, just logical reasoning), there will always be some chance to observe either of the results, even if unlikely. In such a case, your single sample does not "prove" anything, as it could be pure luck. However, if you repeated the experiment many times and saw the same result repeated, it gets less likely that it was luck.
A: In frequentist statistics this is a matter of $\alpha$ (significance level) and $1-\beta$ (power) of a hypothesis test. Usually you would fix $\beta$ (for ex. $80 \%$) and then determine the number of samples required to obtain such power for a desired $\alpha$ (for ex. $0.05$).
A specific example: power analysis for a simple t-test given $d = 1$, $1-\beta=0.8$, and $\alpha=0.05$ returns a required sample size of $n \approx 17$. Increasing $1-\beta=0.9$ yields $n \approx 22$. The significance level has stayed the same in both cases, i.e. the CI obtained from such tests will cover the true values with the same proportions, what has changed is the power, the second test has more power than the first (and narrower CI's).
A: A comment made by "Josh" under this Andrew Gelman's blog post:
https://statmodeling.stat.columbia.edu/2022/01/24/how-large-a-sample-size-does-he-actually-need-he-got-statistical-significance-twice-isnt-that-enough/
helped me (finally!) understand this issue. I copied the comment here, hope that's OK:
"Let’s say we take 100 samples under the null distribution and 100 samples under the alternative distribution. We expect ~5 samples under the null to be significant. If we have power of 80 percent, we’d expect ~80 significant results under the alternative. Here only 5/85 of our results are false positives.
However, what if we have statistical power of only 10 percent. Then we expect ~10 samples under the alternative to be significant. Here 5/15 of our significant results would be false positives.
Decreasing power increases the likelihood that we are in situation one (significant result with the null) rather than in situation two (significant result with the alternative)."
