Why is significance level called the probability of rejecting the null hypothesis given that the null hypothesis is true?
This is my understanding of hypothesis testing, correct me if I'm wrong. For example, the p-value tells us what's the probability of obtaining this or even more extreme value of given statistic. If it turns out to be, say, 0.1%, it's very unlikely that this value of the statistic was random, happened without reason. Null hypothesis is used to sort of justify the alternative hypothesis, right? We show that it's very unlikely this might happen given null hypothesis is true, that's why it justifies the alternative hypothesis.
Now, the question is what is likely or not and this is the significance level we choose. If we chose it to be 1%, then given p-value equal 0.1% we're going to reject the null hypothesis. Why is significance level equal the probability of rejecting the null hypothesis, given it's true? Because it's the probability we can actually get such a p-value that's more extreme that the significance level we've chosen?