Suppose we have compare the value of a continuous variable $x$ between two groups, A and B, of $n$ samples each. The true distributions of $x$ in $A$ and $B$ are $N(\mu_A,1)$ and $N(\mu_B,1)$ respectively, with $\mu_B>\mu_A$. We perform a two-sample test with confidence level $\alpha$ using the null hypothesis $H_0$ that $\mu_A=\mu_B$.
My friend asked me to find the probability that we fail to reject $H_0$. Wouldn't this just be $\alpha$?
The two-sample "t test" in this case with known variances is really a z test because under the null hypothesis the test statistic
$$t = \frac{\bar{x}_B - \bar{x}_A}{\sqrt{1/n+1/n}} = \sqrt{\frac{n}{2}}\, (\bar{x}_B-\bar{x}_A)$$
has a standard Normal distribution. For a two-tailed test, its critical region therefore consists of all values $t$ for which
$$|t| \gt |\Phi^{-1}(\alpha/2)|$$
(writing $\Phi^{-1}$ for the standard Normal quantile function).
Let $\bar x_A$ and $\bar x_B$ be the two sample means. Your assumptions imply they independently have Normal distributions with common variance $1/n$ and means $\mu_A$ and $\mu_B,$ respectively. Therefore $t$ has a Normal distribution with mean $\sqrt{n/2}\,(\mu_B - \mu_A)$ and its variance is
$$\operatorname{Var}(t) = \operatorname{Var}\left(\sqrt{\frac{n}{2}}\, (\bar{x}_B-\bar{x}_A)\right) = \frac{n}{2}\left(\frac{1}{n} + \frac{1}{n}\right) = 1.$$
Thus, the random variable
$$Z = t - \sqrt{n/2}\,(\mu_b - \mu_A)$$
has a standard Normal distribution.
The chance of rejecting the null is the chance that $t$ lies in the critical region; in terms of $Z$ this means
$$Z \lt \Phi(\alpha/2)- \sqrt{n/2}\,(\mu_B - \mu_A) \text{ or } Z \gt \Phi(1-\alpha/2)- \sqrt{n/2}\,(\mu_B - \mu_A)$$
and because the distribution function of $Z$ is $\Phi$ and $Z$ is a continuous random variable, this can be expressed as
$$\eqalign{ \Pr(\text{reject }H_0) &= \Phi\left(\Phi^{-1}(\alpha/2)- \sqrt{n/2}\,(\mu_B - \mu_A)\right) \\&+ 1 - \Phi\left(\Phi^{-1}(1-\alpha/2)- \sqrt{n/2}\,(\mu_B - \mu_A)\right).}\tag{*}$$
A particularly simple case occurs when $H_0$ holds: that is, $\mu_A - \mu_B = 0,$ for then the probability simplifies to
$$\Phi\left(\Phi^{-1}(\alpha/2)\right) + 1 - \Phi\left(\Phi^{-1}(1-\alpha/2))\right) = \alpha/2+1 - (1-\alpha/2)=\alpha,$$
as intended: the chance of rejecting the null when the null holds is the test size $\alpha.$
Generally, since you are considering the case $\mu_B\ge \mu_A,$ the first term in $(*)$ is less than $\alpha/2$ (and decreasing very rapidly as the difference of means increases) but the second term more than makes up for that, increasing quickly. Here is a plot of the chance of rejecting the null against the difference of means for sample size $n=10$ and test size $\alpha=0.05:4
The points are the actual rejection rates in 21 independent simulations of 10,000 pairs of datasets each: they fall along the curve, as one would hope.
This is the R code that produced the figure, written to parallel the analysis in this post.
#
# Theory: `f` is the power, `delta` is mu_B - mu_A, `n` is sample size.
#
f <- function(delta, n, alpha) {
z <- qnorm(alpha/2)
pnorm(z - sqrt(n/2)*delta) + pnorm(-z - sqrt(n/2)*delta, lower.tail=FALSE)
}
alpha <- 0.05
n <- 10
delta.max <- sqrt(2/n) * (3 - qnorm(alpha/2))
curve(f(x, n, alpha), 0, delta.max, ylim=0:1, lwd=2,
main=expression(paste("Chance of Rejecting ", H[0])),
xlab=expression(mu[B] - mu[A]),
ylab="Probability")
abline(h=alpha, lty=3, col="Gray")
#
# Simulation: `delta` is mu_B-mu_A; `n` is sample size, `N` is # of replications
#
sim <- function(delta, n, N, alpha) {
x <- matrix(rnorm(n*N), n)
y <- matrix(rnorm(n*N, delta), n)
z <- (colMeans(y) - colMeans(x)) * sqrt(n/2) # The test statistic
mean(abs(z) > -qnorm(alpha/2)) # Empirical rate of rejection
}
delta <- seq(0, delta.max, length.out=21)
p <- sapply(delta, sim, n=n, N=1e4, alpha=alpha)
points(delta, p, pch=21, bg="Red")
whuber's answer is good, but I think I can simplify things a little bit.
When $\mu_A = \mu_b$, the probability we reject $H0$ is simply the type one error $\alpha$. This is true because $H0$ is indeed the truth.
When $\mu_A \neq \mu_b$, the probability of rejecting the null hypothesis is known as statistical power or sometimes just power. If the population variances are known, the two sample test reduces to a z-test. Let's assume we don't know the population variance, as would be the case for many real life examples. Let's use formulae for the t test.
The power (that is, the probability to reject the null when the alternative is the truth) is given by
$$ 1-\beta = 1- \Phi( \Phi^{-1}(z_{1-\alpha/2} - \delta \cdot 0.5 \cdot \sqrt{N}))$$
Here
$\beta$ is called the "false negative rate" (i.e. the probability of failing to reject the null when it is true). Thus, $1-\beta$ is the probability of rejecting the null when it is false, also known as the power.
$\Phi$ is the CDF if the standard normal, which means $\Phi^{-1}$ is the inverse CDF.
$z_{1-\alpha/2}$ is the critical value. When $\alpha = 0.05$ then this is 1.96. This assumes we are interested in a two tailed test.
$\delta = \vert \mu_a - \mu_b \vert $
The factor of 0.5 is interesting. The formulae I present here are actually used to calculate power for linear regression (which makes this a power calculation for the t-test, as I've mentioned). This formula uses the standard deviation of the predictor to compute power. If we were to do a t test with regression, we would need a binary indicator as the covariate we are regressing on. If the two groups each have size $n$ (that is, they have the same size), then the standard deviation of the binary predictor is $\sigma = \sqrt{0.5^2} = 0.5$. This is where this factor comes from.
$N$ is the total sample size. If each group has size $n$ then $N = 2n$.
Let's plot the power as calculated from this formula as well as the simulated power. Here is some R code:
set.seed(0)
sim_power = function(delta){
p = replicate(10000,{
# Note, the means are the same
a = rnorm(10)
b = rnorm(10, delta)
test = abs(mean(a) - mean(b))/(sqrt(1/10 + 1/10))
# Assume we call results below 0.05 "signigicant"
test>qnorm(0.975)
})
mean(p)
}
delta = seq(0,2.0, 0.01)
p = purrr::map_dbl(delta, sim_power)
plot(delta,p, type = 'l')
calc_power = 1-pnorm( qnorm(0.975) - delta*0.5*sqrt(20) )
lines(delta, calc_power, col='red')
Which produces the following plot (note, the formula is colored red in this plot and the simulated values are colored black)
Save for very small differences, the formula for the power of the t-test looks very similar to the simulated power! And this is for sample sizes as small as 20 (10 in each group).
So, to answer the question of "what is the probability we fail to reject $H0$ [when it is false]" the answer is "it depends" but for the example you've provided, the probability as a function of the difference between means is
$$ \Phi( \Phi^{-1}(z_{1-\alpha/2} - \delta \cdot 0.5 \cdot \sqrt{N}))$$
THE GOOD NEWS
This is what a power calculation gives you, such as in the pwr package in R
THE BAD NEWS
A power calculation requires you to specify a minimum effect size of interest, in addition to sample sizes and an $\alpha$-level.$^{\dagger}$ You then are able to say something like, "With 15 observations in each group and $\alpha=0.05$, I have a 25% chance of failing to catch a situation where the true difference is 1," where 1 is that minimum effect size of interest. (The power is 75%, hence a 25% chance of failing to reject.)
I do not know of a good post on here about calculating power, but there must be one. I do like JBStatistics: https://www.youtube.com/watch?v=NbeHZp23ubs&pbjreload=10.
$^{\dagger}$ You also would specify if it is a one-sided or two-sided test, though the assumption is a two-sided test unless it is otherwise specified.
