Is there a test that uses $|{\mu_A}-{\mu_B}|\le \delta $ as the null hypothesis?

Question

In the usual t-test, the null hypothesis is that "the difference between the means of the two groups is zero".

My Question:

Is there a test that uses "the difference between the means of the two groups is less than a certain value" as the null hypothesis?

"The difference between the means of the two groups is less than a certain value", means that the mean of groups A and B satisfies $$|{\mu_A}-{\mu_B}|\le \delta .$$ So, $$H_0: |{\mu_A}-{\mu_B}|\le \delta $$ and $$H_1: |{\mu_A}-{\mu_B}|> \delta $$

Here, ${\mu_A}$ and ${\mu}_{B}$ are the population mean of groups A and B, respectively and the $\delta >0$ is a predetermined real number.

The population and sample population can be assumed to meet the same requirements as the t-test. If necessary, you can use following settings;

• The populations of both Group A and Group B follow the normal distribution.
• The mean, unbiased SD, sample size of group A, calculated from observed data are: $m_A , s_A, n_A$
• The mean, unbiased SD, sample size of group B, calculated from observed data are: $m_B , s_B, n_B$
• The　$t_{obs}$ represents the t-value calculated from observed data.

Furthermore, if necessary, the population variance of both groups can be considered to be equal. Then the pooled sd is as follows. $${s^*}=\sqrt{\frac{({n_A}-1){s_A}+({n_B}-1){s_B}}{{n_A}+{n_B}-2}}$$

The $\tau_{\phi ,\mu}$ represents the cumulative Noncentral t-distribution with non centrality parameter $\mu$ and degrees of freedom.
The $\tau_{\phi ,\mu}(t)$　is the value obtained by definite integration of this non-central distribution over the interval from -∞ to t.

Answer 1

You can use the equivalence between confidence intervals and hypothesis testing: Can we reject a null hypothesis with confidence intervals produced via sampling rather than the null hypothesis? Then you will compute the confidence interval for the difference of the means and reject the null hypothesis when none of the values between $\pm \delta$ are inside the interval.

But with this method you will reject the null hypothesis less often than the aimed significance level (ie. the p-value is overestimates, too high, and the power of the test is less). This difference arrises because confidence intervals relate to point hypotheses, which is not your case.

Graphical view of the sample distribution of $\bar{x}-\bar{y}$ and $\hat{\sigma}$

The images below sketch two situations for a t-test:

1. When we compare two samples with equal size and variance and the null hypothesis is $$H_0: \mu_y-\mu_x = 0$$ then we look at the value of the t-statistic, which relates to the likelihood-ratio. $$t = \frac{1}{\sqrt{2/n}} \frac{d}{s_p}$$
2. When we use instead the null hypothesis $$H_0: \vert \mu_y-\mu_x \vert \leq \delta$$ then the likelihood ratio test will work out the same and be like the t-statistic but now the boundaries are shifted to the left and the right.

In the image below the boundaries for the t-value of a 95% significance test are drawn. These boundaries are compared with sample distributions of the standard deviation and difference of means for samples of size 5. The $X$ and $Y$ are normal distributed with equal variance and equal means, except in the lower image where the means differ by $\mu_y-\mu_X = 0.5$.

Likelihood ratio test, T-test with shifted boundaries, not ideal

In the first image, you see that 5% of the samples lead to a rejection of the hypothesis (as designed by setting the level at 95%). However, in the lower image, the rejection rate is lower and not equal to 5% (Because the boundaries are wider due to the shift $\delta$), and the nominal p-value of 5% overestimates the actual p-value.

So possibly one can choose to draw the boundaries more narrow. But for large $s_p$ you get closer to the current boundaries (Intuitively you can say that $\delta$ becomes less important, relatively smaller, when the variance of the variables is large).

The reason is that we do not need to necessarily use the likelihood ratio test, is that we are not dealing with a simple hypothesis. According to the Neyman-Pearson lemma the likelihood ratio test is the most powerful test. But, that is only true when the hypotheses are simple hypotheses (like $H_0: \mu_y-\mu_x = 0$), and we have a composite hypothesis (like $H_0: -\delta \leq \mu_y-\mu_x \leq \delta$). For a composite hypothesis the likelihood ratio test may not always give the specified significance level (we choose boundaries for the likelihood ratio according to the worst case).

So we can make sharper boundaries than the likelihood ratio test. However, there is no unique way to do this.

R-code for the images:

nsim <- 10^4
nsmp <- 5

rowDevs <- function(x) {
  n <- length(x[1,])
  sqrt((rowMeans(x^2)-rowMeans(x)^2)*n/(n-1))
}

### simulations
set.seed(1)
x <- matrix(rnorm(nsim*nsmp),nsim)
y <- matrix(rnorm(nsim*nsmp),nsim)

### statistics of difference and variance
d <- rowMeans(y)-rowMeans(x)
v <- (0.5*rowDevs(x)+0.5*rowDevs(y))

## colouring 5% points with t-values above/below qt(0.975, df = 18)
dv_slope <- qt(0.975, df = 18)*sqrt(2/nsmp)
col <- (d/v > dv_slope)+(d/v < -dv_slope)

### plot points
plot(d,v, xlim = c(-4,4), ylim = c(0,1.5),
     pch = 21, col = rgb(col,0,0,0.1), bg = rgb(col,0,0,0.1), cex = 0.5,
     xlab = expression(d == bar(y)-bar(x)),
     ylab = expression(s[p] == sqrt(0.5*s[x]+0.5*s[y])),
     xaxs = "i", yaxs = "i",
     main = expression(H[0] : mu[y]-mu[x]==0))

lines(c(0,10),c(0,10)/dv_slope, col = 1, lty = 2)
lines(-c(0,10),c(0,10)/dv_slope, col = 1, lty = 2)



## colouring 5% points with t-values above/below qt(0.975, df = 18)
dlt <- 0.5
## colouring 5% points with t-values above/below qt(0.975, df = 18)
dv_slope <- qt(0.975, df = 18)*sqrt(2/nsmp)
col <- ((d-2*dlt)/v > dv_slope)+((d)/v < -dv_slope)

### plot points
plot(d-dlt,v, xlim = c(-4,4), ylim = c(0,1.5),
     pch = 21, col = rgb(col,0,0,0.1), bg = rgb(col,0,0,0.1), cex = 0.5,
     xlab = expression(d == bar(y)-bar(x)),
     ylab = expression(s[p] == sqrt(0.5*s[x]+0.5*s[y])),
     xaxs = "i", yaxs = "i",
     main = expression(H[0] :  "|" * mu[x]-mu[y] * "|" <= delta))

lines(c(0,10)+dlt,c(0,10)/dv_slope, col = 1, lty = 2)
lines(-c(0,10)-dlt,c(0,10)/dv_slope, col = 1, lty = 2)

Why does the t-test work for point hypothesis, $H_0 : \mu = 0$, but not for a composite hypothesis $H_0: \sigma \leq \mu \leq \sigma$?

In the image below we draw the situation like above, but now we change the standard deviation $\sigma$ of the population from which we draw the sample. Now the image contains two separate clouds. In the one case $\sigma = 1$ like before. In the other case $\sigma = 0.2$, and this creates the additional smaller little cloud of points.

The diagonal lines are the borders for some critical level of the likelihood ratio. The first case (upper image) is for a point null hypothesis $H_0 : \mu = 0$, the second case is for a composite hypothesis $H_0: \sigma \leq \mu \leq \sigma$ (where in this particular image $\sigma = 0.15$).

When we consider the probability of rejecting the null hypothesis if it is true (type I error), then this probability will depend on the parameters $\mu$ and $\sigma$ (which can differ within the null hypothesis).

• Dependency on $\mu$: When $\mu$ is closer to either $\pm \delta$ instead of $0$ then it might be intuitive that the null hypothesis is more likely to be rejected, and that we can not make a test such that the the type 1 error is the same for whatever value of $\mu$ that corresponds to the null hypothesis.

• Dependency on $\sigma$: The rejection probability will also depend on $\sigma$.

  • In the first case/image (point hypothesis), then independent of $\sigma$ the type I error will be constant. If we change the $\sigma$ then this relates to scaling the sample distribution (represented by the cloud of points in the image) in both vertical and horizontal directions and the diagonal boundary line will intersect the same proportion.

  • In the second case/image (composite hypothesis), then the the type I error will depend on $\sigma$. The boundary lines are shifted and do not pass through the center of the scaling transformation, so the scaling won't be an invariant transformation anymore with regards to the type I error.

While these borders relate to some critical likelihood ratio, this is based on the ratio for a specific case out of the composite hypotheses, and may not be optimal for other cases. (in the case of point hypotheses there are no 'other cases', or in the case of the "point hypothesis" $\mu_a - \mu_b = 0$, which is not really a point hypothesis because $\sigma$ is not specified in the hypothesis, it happens to work out because the likelihood ratio is independent of $\sigma$).

Answer 2

Alternatively using simulations i.e. the bootstrap method (R code follows).

# Generate 1000 random standard normal values for x and y
x = rnorm(1000,0,1)
y = rnorm(1000,1,1)

# Repeat many times: sample with replacement x and y,
# calculate the mean of the new samples, take the difference
res = replicate(1e4, mean(sample(x,replace=T)) - mean(sample(y,replace=T)))

# Estimate the desired probability
mean(abs(res) <= 1)
[1] 0.1583
mean(abs(res) <= 1.1)
[1] 0.8875

Answer 3

I think one possible solution for this test is to

1. turn to regression to get the two means
2. calculate the absolute value of their difference from the regression coefficients (a non-linear combination). Let's call this random variable $|\Delta|$.

Once this is done, you have two choices.

You can look at the one-sided CI for $|\Delta|$ to see if it excludes your superiority threshold. You can get that easily from step (2), since the overlap between two one-sided 95% CIs makes a two-sided 90% CI, so you can work backwards from the usual 90% CI for $|\Delta|$.

Alternatively, you can perform a two-sided hypothesis test on $|\Delta|$, and then calculate the one-sided p-value from that. This is a bit more work, but is just a matter of getting the sign of the inequality, a $\chi^2$ statistic from the two-sided test, and evaluating cumulative standard normal distribution. If your test returns an F-statistic, you will have to use that instead, along with the t distribution in place of the normal. If you don't want to go this route, when $|\Delta| - \delta$ is positive, you can simply divide the two-sided p-value by 2. In the other case, you need to calculate $1-\frac{p}{2}$ since you are in the other tail. This simpler division approach works for symmetric distributions only.

Here is an example in Stata, where we will conduct two such hypotheses comparing the average price of foreign (foreign = 1) and domestic cars (foreign = 0):

  . sysuse auto, clear
(1978 Automobile Data)

. table foreign, c(mean price)

-----------------------
 Car type | mean(price)
----------+------------
 Domestic |     6,072.4
  Foreign |     6,384.7
-----------------------

. /* (1) Calculate the means using regression */
. regress price ibn.foreign, noconstant

      Source |       SS           df       MS      Number of obs   =        74
-------------+----------------------------------   F(2, 72)        =    159.91
       Model |  2.8143e+09         2  1.4071e+09   Prob > F        =    0.0000
    Residual |   633558013        72  8799416.85   R-squared       =    0.8162
-------------+----------------------------------   Adj R-squared   =    0.8111
       Total |  3.4478e+09        74  46592355.7   Root MSE        =    2966.4

------------------------------------------------------------------------------
       price |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
     foreign |
   Domestic  |   6072.423    411.363    14.76   0.000     5252.386     6892.46
    Foreign  |   6384.682   632.4346    10.10   0.000     5123.947    7645.417
------------------------------------------------------------------------------

. /* (2) Calculate the absolute value of the foreign-domestic difference */
. nlcom av_diff:abs(_b[1.foreign] - _b[0.foreign]), level(90) post

     av_diff:  abs(_b[1.foreign] - _b[0.foreign])

------------------------------------------------------------------------------
       price |      Coef.   Std. Err.      z    P>|z|     [90% Conf. Interval]
-------------+----------------------------------------------------------------
     av_diff |   312.2587   754.4488     0.41   0.679    -928.6992    1553.217
------------------------------------------------------------------------------

. /* (3a) We know that a one-sided 95% CI is (-inf,1553.217] */
. /* (3b) Transform two-sided test into a one-sided test and get p-values */
. // Test something just inside the CI */
. // H_0': (avg_price_foreign - avg_price_domestic) <= 1553
. // H_1': (avg_price_foreign - avg_price_domestic)  > 1553
. test av_diff = 1553

 ( 1)  av_diff = 1553

           chi2(  1) =    2.70
         Prob > chi2 =    0.1001

. local sign_av_diff = sign(_b[av_diff] - 1553) // get the sign

. display "p-value' = " normal(`sign_av_diff'*sqrt(r(chi2)))
p-value' = .05002962

. // Test something just above the CI */
. // H_0'': (avg_price_foreign - avg_price_domestic) <= 1554
. // H_1'': (avg_price_foreign - avg_price_domestic)  > 1554
. test av_diff = 1554

 ( 1)  av_diff = 1554

           chi2(  1) =    2.71
         Prob > chi2 =    0.0998

. local sign_av_diff = sign(_b[av_diff] - 1554) // get the sign

. display "p-value = " normal(`sign_av_diff'*sqrt(r(chi2)))
p-value = .049893

The one-sided 95% CI is $(-\infty, 1553.217]$, so $\delta>1553.217$ in order of us to reject. If we try testing a value below that upper bound like 1553, the one-sided p-value is .05003, so we cannot reject. If we test something just above the UB, like 1554, the p-value is .049893, so we can reject at $\alpha=5\%$. I don't advocate using rigid thresholds for significance, this is just meant to illustrate the intuition. Note that you can also divide the two-sided p-values by 2 to get this (Stata's two-sided p-values are on the "Prob > chi2" line).

Here the null is $H_0=|\Delta|\le \delta$ (practical equivalence) versus $H_a=|\Delta| > \delta$ (non-equivalence). We focus on testing $|\Delta| = \delta$, so we calculate the probability at the most extreme point of the null hypothesis, closest to alternative parameter space. This means that the p-value is exact only for $|\Delta| = \delta$. If $|\Delta| < \delta$, then our p-value is just a conservative bound on the type I error rate (the error being finding a negative effect when there is none).

Answer 4

You can perform a t-test and just look at confidence intervals. In some circumstances (e.g. clinical trials) you are not interested in statistical significance, but whether the difference is significant from a practical point of view by adding a margin $\delta$ (in a clinical trials setting it’s called clinical significance). Have a look at the picture. We assess mean response difference in experimental and control group.

Answer 5

one of ideas is to add $\delta$ to one population (raising mean) and in second test substracting $\delta$ and then computing statistic and figure out in two "one-sided tests" p-values, after adding these you will have one p-value for two sided test stated in your question

it's like solving equation in elementary school:

$$|\mu_A - \mu_B| \le \delta => \begin{cases} \mu_A - \mu_B \le \delta, & \text{if}\ \mu_A - \mu_B \ge 0 \\[2ex] \mu_A - \mu_B \ge -\delta, & \text{if}\ \mu_A - \mu_B < 0 \end{cases} =>\begin{cases} (\mu_A-\delta) - \mu_B \le 0, & \text{if}\ \mu_A - \mu_B \ge 0 \\[2ex] (\mu_A+\delta) - \mu_B \ge 0, & \text{if}\ \mu_A - \mu_B < 0 \end{cases} =>\begin{cases} (\mu_A-\delta) \le \mu_B, & \text{if}\ \mu_B \le \mu_A\\[2ex] (\mu_A+\delta) \ge \mu_B, & \text{if}\ \mu_B > \mu_A \end{cases}$$ this is your $H_0$ :) now let's construct $H_1$ $$H_0\begin{cases} (\mu_A-\delta) \le \mu_B, & \text{if}\ \mu_B \le \mu_A\\[2ex] (\mu_A+\delta) \ge \mu_B, & \text{if}\ \mu_B > \mu_A \end{cases}, H_1\begin{cases} (\mu_A-\delta) \ge \mu_B, & \text{if}\ \mu_B \le \mu_A, & (1)\\[2ex] (\mu_A+\delta) \le \mu_B, & \text{if}\ \mu_B > \mu_A, & (2) \end{cases}$$

for $(1)$ you want to compute p-value that $$p((\mu_A-\delta) \ge \mu_B|\mu_A \ge \mu_B) = \frac{p((\mu_A-\delta) \ge \mu_B)}{p(\mu_A \ge \mu_B)}$$ analogous for $(2)$,

and combining $$p(|\mu_A - \mu_B| \le \delta) = 1-p((\mu_A-\delta) \ge \mu_B|\mu_A \ge \mu_B) - p((\mu_A+\delta) \le \mu_B|\mu_A \lt \mu_B)$$

ask questions if needed, I am not entirely sure of this approach, and would welcome any critique

Answer 6

Just for a comment;

Let $F$ be the cumulative distribution of $p(\ \ |{H_0})$, that means

$$F(t) = p(t>T |\ {H_0}\ is\ true) . \tag{1-1}$$

Here, $p(t>-\infty\ |\ {H_0}\ is\ true)$ is the　probability that $t>T$ under the condition that $H_0$ is true, $T$ is a random value representing the t-value. The $t$ is a real number substituted to the $F$.

And, let $t_{obs}$ be the t-value calculated from actual observations.

Then, the p-value shall be; $$p-value = p(|t|>|t_{obs}|\ |\ {H_0}\ is\ true). \tag{1-2}$$

Therefore, $$p-value = p(|t|>|t_{obs}|\ |\ {H_0}\ is\ true)$$ $$=p(\ t>|t_{obs}|\ or\ \ t<-|t_{obs}|\ |\ {H_0}\ is\ true)$$ $$=p(\ t>|t_{obs}|\ |\ {H_0}\ is\ true)\ +\ p(\ t<-|t_{obs}|\ |\ {H_0}\ is\ true) $$ $$=F(-|t_{obs}|)+(1-F(|t_{obs}|))$$ $$=1+F(-|t_{obs}|)-F(|t_{obs}|) \tag{1-3}$$

Thus, the essence of my question would be what function $F$ in (1-1) would be under my ${H}_{0}$.

If the mean and standard deviation of the population are known, I think these distributions can be brought to a form similar to the simulation of user2974951 by using the regenerability of the normal distribution.

However, if both of the mean and standard deviation of the population are unknown, then I have no idea.

I'm waiting for your opinion.