Does rejecting a study's credibility based on sample size make sense even if statistical significance is reached? I've seen people do this commonly, and to me, it makes no sense because one of the main reasons of having a large enough sample size is that you CAN reach statistical significance in the first place.
 A: This is to some extent answered in this paper. What it describes is a setting, where

*

*We (sort of randomly) pick research hypothesis to investigate, for which the null hypothesis is either exactly true or we have the hypothesized effect size.

*Of course, we don't know for sure when we start our study.

*In that setting, a "small sample size means smaller power and [...] the PPV [positive preditive value] for a true research finding" when observing $p\leq 0.05$ decreases as power decreases.

*I.e. how likely it is that $p\leq 0.05$ (or any other threshold
$\alpha$) indicates a correct rejection of the null hypothesis
depends on the power. The other thing that matters is what proportion
of investigated alternative hypotheses are true (or in practice, how
implausible your research finding is a-priori).

*This means surprising findings with $p\leq \alpha$ from small studies are much less likely to be true than expected findings from very large studies.

*The
example in the essay is overly simplified, because few point null
hypotheses are exactly true and even the meaningfully large effects
follow some kind of distribution, but the underlying point is true.

Why does this happen? Well, the p-value is random variable that is in turn based on the other random variables in your study. The smaller the sample size, the noisier the p-value is. Consider an extreme case:

*

*Let's assume we have absolutely no data.

*A valid null hypothesis test in that setting is to reject the null hypothesis
randomly with probability $\alpha$.

*If we know nothing else, then the probability that a study with $p\leq \alpha$ has produced a true research finding is then simply the a-priori probability that the alternative hypothesis was true.

You may think I'm being silly here to do a null hypothesis test without any data, but when I do a study with a power that is really low (=I have hardly any data), then I am essentially doing the same thing (i.e. looking at a random variable that has very little connection with my research question). That's what the paper I linked pointed out.
