I often see the p-value being defined as "A p-value can be defined as the probability of obtaining a test result, or a more extreme one, given that H0 is true". When we define our significance level, alpha (usually 0.05), we're essentially trying to say that there's a 5% risk that we will incorrectly reject it (commiting a type 1 error).
Now, if the goal of many testes (besides normality, etc) is to reject the H0 (p-value < alpha), and given that alpha tells us the risk that we will incorrectly reject H0, how certain are we that when rejecting the H0, we're actually seeing difference in our samples and the samples are not at random?
Let's say population A and B with the same mean of 170. Essentially, H0 would be true, but we could very well take a group of individuals from population A that have a mean of 150 and another group from population B that have a mean of 180 and the resulting test statistic could be high enough so that the p-value would be < than alpha, thus leading to the rejection of H0, although the populations have the same mean (H0: uA = uB)
In other words, if alpha = 0.05, we know that if H0 is true, there’s a 5% risk that we will incorrectly reject it (commiting a type 1 error if H0 is true). So we want the risk of commiting this type 1 error (if H0 is true) to be less than 5% and only then do we accept our test as statistical significant?
I hope I'm making sense and what I said was (partly) correct. I'm just trying to wrap my head around how p-value is actually useful (given that we can very well have a significant value even though it's false - I believe this would be quite bad in clinical trials and I'm assuming the alpha would be even less)
Thank you