what does For a significance threshold of 0.05, if the null hypothesis is true mean Although it's a simple question and I know what the null hypothesis means, I couldn't understand this sentence.
For a significance threshold of 0.05, if the null hypothesis is true, then the probability of correctly failing to reject the null is 1 – 0.05 = 0.95.
Can someone please explain?
 A: If the null is true it means that, for example, there is no difference between the distribution of population X and population Y.
Imagine you take a sample from X and a sample from Y and you do a t-test for difference between the two. If you repeat this sampling and testing many times, you will find that the t-test returns p > 0.05 in 95% of the cases. I.e. the null has not being rejected 95% of the time and this the "correct" outcome.
This simulation in R may help. Hopefully, you can understand it even if you don't use R. We compare a sample from population x and y 1000 times, apply a test and check how many times we get p < 0.05:
pval <- rep(NA, 1000) 

for(i in 1:length(pval)) {
    set.seed(i)
    x <- rnorm(n= 10, mean= 0, sd= 1)
    y <- rnorm(n= 10, mean= 0, sd= 1)
    pval[i] <- t.test(x, y)$p.value
}

sum(pval < 0.05) # 50 times p < 0.05

We can also show that if the null is true, p-values follow a uniform distribution:
hist(pval, breaks= seq(0, 1, by= 0.05))


(correctly failing to reject is a bit contrived, I think.)
A: Frequentist statistics (what you’re doing when you use a p-value) puts controls on how often you will make a fool out of yourself and assert something incorrect.
Traditionally, people have considered it a major error to assert that a null hypothesis is false when it isn’t. Setting $\alpha=0.05$ means that there is only a $1$ in $20$ chance of making such an error when the null is true.$^{\dagger}$
$$
P(\text{ reject }\vert \text{ }H_0\text{ is true })=0.05
$$
So you make mistakes of this sort fairly infrequently.
If you like thinking of sensitivity and specificity, the test has $95\%$ specificity (and then power is sensitivity).
$^{\dagger}$ This assumes that the hypothesis test has proper calibration (t-test does) and that the assumptions are met. More exotic hypothesis tests might not have calibration as good as the t-test. If you get into such tests, you will have to make a judgment call about if it is acceptable to have a $6\%$ or $8\%$ chance of making such a false rejection.
A: A p-value provides the probability of receiving a particular test statistic or a more extreme one given all of the p-value's assumptions hold true. One of these assumptions is that the null hypothesis (often "of no effect") is indeed true. Given all assumptions are met, including that the null hypothesis is true, and an infinite number of trials, a result with p ≤ .05 is expected to happen only in 5% (.05) of trials (as also visible in the uniform p-value distribution of @dariober's plot).
If you set a significant threshold/level of .05, you consider every result/test statistic with a p-value < .05 as "significant" and reject the null hypothesis. Now if in a significance test the null hypothesis is in fact true and you receive p < .05, you'd incorrectly reject the null hypothesis in these cases. And since by definition the probabilities of disjunct events sum to 1, you correctly do not reject the null hypothesis in 1 - 0.05 = 0.95 = 95% of trials/samples (in the long run, i.e. given an infinite number of repetitions).
