Repeated testing to increase confidence?

Question

I want to know if the population mean equals some value $m$. For this puporse, I am using a one-sample t-test.

The null-hypothesis of the one-sample t-test is that the difference between population mean $\mu$ and given value $m$ is different using the sample mean $\overline{x}$.

Doing the t-test onces using a random sample gives me a t-test statistic and some p-value. The p-value might or might not be significant (p<0.05), depending on the random sample.

In order to be more confident about the test, wouldn't it be useful to repeat it several times? For instance, I could draw 100 random samples, perform for each sample the t-test and then report summary statistics for the t-test statistics and p-values. If all p-values are below 0.05 then I would be much more confident that I can safely reject the null hypothesis.

However, I have never seen somone doing this kind of repeated testing. Is this approach not valid?

Answer 1

When the test is performed once, the results of the t-test already sum up what would happen across a very large number of hypothetical repetitions of your study under the assumption that the null hypothesis is true. This type of hypothetical repetition of a study allows you to take into account the sampling variability (i.e., the notion that different samples of the same size selected at random from the same population are expected to give different results).

But you could repeat the study a small number of times in real life under similar conditions - then you would venture into a different territory, called "replication". In that case, you would have to determine whether your study is important enough to warrant replication, etc. See https://www.verywellmind.com/what-is-replication-2795802 for a brief intro to replication.