In the usual t-test, the null hypothesis is that "the difference between the means of the two groups is zero".

My Question:

Is there a test that uses "the difference between the means of the two groups is less than a certain value" as the null hypothesis?

"The difference between the means of the two groups is less than a certain value", means that the mean of groups A and B satisfies $$|{\mu_A}-{\mu_B}|\le \delta .$$ So, $$H_0: |{\mu_A}-{\mu_B}|\le \delta $$ and $$H_1: |{\mu_A}-{\mu_B}|> \delta $$

Here, ${\mu_A}$ and ${\mu}_{B}$ are the population mean of groups A and B, respectively and the $\delta >0$ is a predetermined real number.

The population and sample population can be assumed to meet the same requirements as the t-test. If necessary, you can use following settings;

  • The populations of both Group A and Group B follow the normal distribution.
  • The mean, unbiased SD, sample size of group A, calculated from observed data are: $m_A , s_A, n_A$
  • The mean, unbiased SD, sample size of group B, calculated from observed data are: $m_B , s_B, n_B$
  • The $t_{obs}$ represents the t-value calculated from observed data.

Furthermore, if necessary, the population variance of both groups can be considered to be equal. Then the pooled sd is as follows. $${s^*}=\sqrt{\frac{({n_A}-1){s_A}+({n_B}-1){s_B}}{{n_A}+{n_B}-2}}$$

The $\tau_{\phi ,\mu}$ represents the cumulative Noncentral t-distribution with non centrality parameter $\mu$ and degrees of freedom.
The $\tau_{\phi ,\mu}(t)$ is the value obtained by definite integration of this non-central distribution over the interval from -∞ to t.

  • $\begingroup$ Are you just looking for a one-sided test of superiority? $\endgroup$
    – dimitriy
    Oct 26, 2020 at 6:23
  • $\begingroup$ @Dimitriy V. Masterov What the "test of superiority" you mean? Is this a kaind of the TOST? I found the following PDF with the key words superiority testhttps://ncss-wpengine.netdna-ssl.com/wp-content/themes/ncss/pdf/Procedures/NCSS/Two-Sample_T-Test_for_Superiority_by_a_Margin.pdf $\endgroup$ Oct 26, 2020 at 6:39
  • $\begingroup$ @ Dimitriy V. Masterov Indeed. Is it like the superiority test on the PDF file I mentioned combined with a method like the TOST? $\endgroup$ Oct 26, 2020 at 6:48
  • 2
    $\begingroup$ Why isn’t the standard TOST adequate? $\endgroup$
    – Dave
    Oct 26, 2020 at 11:01
  • 2
    $\begingroup$ @Dave a TOST is not exactly the same question. See the image in treskov's answer. It is a test for a null hypothesis that means 'no equivalence' $$H_0: |{\mu_A}-{\mu_B}|\ge \delta$$ and when this is rejected you say the effect 'equivalence' is shown, and otherwise the null 'no equivalence' not rejected (or equivalence is not shown). $\endgroup$ Oct 26, 2020 at 13:23

6 Answers 6


You can use the equivalence between confidence intervals and hypothesis testing: Can we reject a null hypothesis with confidence intervals produced via sampling rather than the null hypothesis? Then you will compute the confidence interval for the difference of the means and reject the null hypothesis when none of the values between $\pm \delta$ are inside the interval.

But with this method you will reject the null hypothesis less often than the aimed significance level (ie. the p-value is overestimates, too high, and the power of the test is less). This difference arrises because confidence intervals relate to point hypotheses, which is not your case.

Graphical view of the sample distribution of $\bar{x}-\bar{y}$ and $\hat{\sigma}$

The images below sketch two situations for a t-test:

  1. When we compare two samples with equal size and variance and the null hypothesis is $$H_0: \mu_y-\mu_x = 0$$ then we look at the value of the t-statistic, which relates to the likelihood-ratio. $$t = \frac{1}{\sqrt{2/n}} \frac{d}{s_p}$$
  2. When we use instead the null hypothesis $$H_0: \vert \mu_y-\mu_x \vert \leq \delta$$ then the likelihood ratio test will work out the same and be like the t-statistic but now the boundaries are shifted to the left and the right.

In the image below the boundaries for the t-value of a 95% significance test are drawn. These boundaries are compared with sample distributions of the standard deviation and difference of means for samples of size 5. The $X$ and $Y$ are normal distributed with equal variance and equal means, except in the lower image where the means differ by $\mu_y-\mu_X = 0.5$.


Likelihood ratio test, T-test with shifted boundaries, not ideal

In the first image, you see that 5% of the samples lead to a rejection of the hypothesis (as designed by setting the level at 95%). However, in the lower image, the rejection rate is lower and not equal to 5% (Because the boundaries are wider due to the shift $\delta$), and the nominal p-value of 5% overestimates the actual p-value.

So possibly one can choose to draw the boundaries more narrow. But for large $s_p$ you get closer to the current boundaries (Intuitively you can say that $\delta$ becomes less important, relatively smaller, when the variance of the variables is large).

The reason is that we do not need to necessarily use the likelihood ratio test, is that we are not dealing with a simple hypothesis. According to the Neyman-Pearson lemma the likelihood ratio test is the most powerful test. But, that is only true when the hypotheses are simple hypotheses (like $H_0: \mu_y-\mu_x = 0$), and we have a composite hypothesis (like $H_0: -\delta \leq \mu_y-\mu_x \leq \delta$). For a composite hypothesis the likelihood ratio test may not always give the specified significance level (we choose boundaries for the likelihood ratio according to the worst case).

So we can make sharper boundaries than the likelihood ratio test. However, there is no unique way to do this.

R-code for the images:

nsim <- 10^4
nsmp <- 5

rowDevs <- function(x) {
  n <- length(x[1,])

### simulations
x <- matrix(rnorm(nsim*nsmp),nsim)
y <- matrix(rnorm(nsim*nsmp),nsim)

### statistics of difference and variance
d <- rowMeans(y)-rowMeans(x)
v <- (0.5*rowDevs(x)+0.5*rowDevs(y))

## colouring 5% points with t-values above/below qt(0.975, df = 18)
dv_slope <- qt(0.975, df = 18)*sqrt(2/nsmp)
col <- (d/v > dv_slope)+(d/v < -dv_slope)

### plot points
plot(d,v, xlim = c(-4,4), ylim = c(0,1.5),
     pch = 21, col = rgb(col,0,0,0.1), bg = rgb(col,0,0,0.1), cex = 0.5,
     xlab = expression(d == bar(y)-bar(x)),
     ylab = expression(s[p] == sqrt(0.5*s[x]+0.5*s[y])),
     xaxs = "i", yaxs = "i",
     main = expression(H[0] : mu[y]-mu[x]==0))

lines(c(0,10),c(0,10)/dv_slope, col = 1, lty = 2)
lines(-c(0,10),c(0,10)/dv_slope, col = 1, lty = 2)

## colouring 5% points with t-values above/below qt(0.975, df = 18)
dlt <- 0.5
## colouring 5% points with t-values above/below qt(0.975, df = 18)
dv_slope <- qt(0.975, df = 18)*sqrt(2/nsmp)
col <- ((d-2*dlt)/v > dv_slope)+((d)/v < -dv_slope)

### plot points
plot(d-dlt,v, xlim = c(-4,4), ylim = c(0,1.5),
     pch = 21, col = rgb(col,0,0,0.1), bg = rgb(col,0,0,0.1), cex = 0.5,
     xlab = expression(d == bar(y)-bar(x)),
     ylab = expression(s[p] == sqrt(0.5*s[x]+0.5*s[y])),
     xaxs = "i", yaxs = "i",
     main = expression(H[0] :  "|" * mu[x]-mu[y] * "|" <= delta))

lines(c(0,10)+dlt,c(0,10)/dv_slope, col = 1, lty = 2)
lines(-c(0,10)-dlt,c(0,10)/dv_slope, col = 1, lty = 2)

Why does the t-test work for point hypothesis, $H_0 : \mu = 0$, but not for a composite hypothesis $H_0: \sigma \leq \mu \leq \sigma$?

In the image below we draw the situation like above, but now we change the standard deviation $\sigma$ of the population from which we draw the sample. Now the image contains two separate clouds. In the one case $\sigma = 1$ like before. In the other case $\sigma = 0.2$, and this creates the additional smaller little cloud of points.

The diagonal lines are the borders for some critical level of the likelihood ratio. The first case (upper image) is for a point null hypothesis $H_0 : \mu = 0$, the second case is for a composite hypothesis $H_0: \sigma \leq \mu \leq \sigma$ (where in this particular image $\sigma = 0.15$).

When we consider the probability of rejecting the null hypothesis if it is true (type I error), then this probability will depend on the parameters $\mu$ and $\sigma$ (which can differ within the null hypothesis).

  • Dependency on $\mu$: When $\mu$ is closer to either $\pm \delta$ instead of $0$ then it might be intuitive that the null hypothesis is more likely to be rejected, and that we can not make a test such that the the type 1 error is the same for whatever value of $\mu$ that corresponds to the null hypothesis.

  • Dependency on $\sigma$: The rejection probability will also depend on $\sigma$.

    • In the first case/image (point hypothesis), then independent of $\sigma$ the type I error will be constant. If we change the $\sigma$ then this relates to scaling the sample distribution (represented by the cloud of points in the image) in both vertical and horizontal directions and the diagonal boundary line will intersect the same proportion.

    • In the second case/image (composite hypothesis), then the the type I error will depend on $\sigma$. The boundary lines are shifted and do not pass through the center of the scaling transformation, so the scaling won't be an invariant transformation anymore with regards to the type I error.

While these borders relate to some critical likelihood ratio, this is based on the ratio for a specific case out of the composite hypotheses, and may not be optimal for other cases. (in the case of point hypotheses there are no 'other cases', or in the case of the "point hypothesis" $\mu_a - \mu_b = 0$, which is not really a point hypothesis because $\sigma$ is not specified in the hypothesis, it happens to work out because the likelihood ratio is independent of $\sigma$).

example with different sigma

  • $\begingroup$ I have a question: you are saying that there is no unique way to compute confidence interval in this situation - sure although you can find most powerful test for one sided hypothesis and I think that this is a way to go (since you brought up Neyman-Pearson) do you think it's correct statement? $\endgroup$
    – quester
    Oct 30, 2020 at 8:15
  • 1
    $\begingroup$ @quester the Neyman Pearson lemma depends on a simple null hypothesis, whereas this is a case with a composite null hypothesis. Indeed, when the alternative hypothesis is one-sided (e.g. $H_a:\mu>\delta$) then there is a unique solution that is most powerful, because the worst case of the composite hypothesis $H_0:−\delta≤\mu≤\delta$ coincides with a point hypothesis $H_0:\mu=\delta$. $\endgroup$ Oct 30, 2020 at 8:45
  • $\begingroup$ Actually, I believe that the lemma also requires the alternative hypothesis to be a point hypothesis, but often it works anyway because the result is the same for every possible alternative point hypothesis inside the composite hypothesis. $\endgroup$ Oct 30, 2020 at 8:46
  • $\begingroup$ sure it's about point hypothesis but I wanted to aim at most powerful test for one-side hypothesis, then having two most powerful one sided tests for $\alpha=0.975$ then we will have one two sided test with $\alpha=0.95$ en.wikipedia.org/wiki/… if I made a mistake just point me to any reading material :) $\endgroup$
    – quester
    Oct 30, 2020 at 9:43
  • 1
    $\begingroup$ @quester that Karlin-rubin theorem is a more formal expression of what I stated with "often it works anyway because the result is the same for every possible alternative point hypothesis inside the composite hypothesis" however... $\endgroup$ Oct 30, 2020 at 9:50

Alternatively using simulations i.e. the bootstrap method (R code follows).

# Generate 1000 random standard normal values for x and y
x = rnorm(1000,0,1)
y = rnorm(1000,1,1)

# Repeat many times: sample with replacement x and y,
# calculate the mean of the new samples, take the difference
res = replicate(1e4, mean(sample(x,replace=T)) - mean(sample(y,replace=T)))

# Estimate the desired probability
mean(abs(res) <= 1)
[1] 0.1583
mean(abs(res) <= 1.1)
[1] 0.8875
  • $\begingroup$ Thank you for your answer. You are calculating the counts the t in the range of $-1<t<1$ on the "hist(res)" am I right? $\endgroup$ Oct 26, 2020 at 10:32
  • $\begingroup$ @BlueVarious In res I have 10 thousand results obtained from bootstrapping, that is 10 thousand mean differences obtained from the original data via random sampling. In the last two lines of code I am calculating the number of cases (counts) which have a mean absolute difference less than or equal to 1, divided by all the cases. So yes, these are the values between -1 and 1 in a histogram (if you take the raw values, so no absolute function). $\endgroup$ Oct 26, 2020 at 10:41
  • $\begingroup$ @BlueVarious Try mean(res >= -1 & res <= 1) for the same result. $\endgroup$ Oct 26, 2020 at 10:46
  • $\begingroup$ So, in the setting of "mean(abs(res) <= 1)", $\delta=1$, ${\mu_A} =1$, and ${\mu_B}=0$, am I right? $\endgroup$ Oct 26, 2020 at 10:46
  • 1
    $\begingroup$ @BlueVarious Not true, try varying the sample size, that is the first argument in x and y (1000 in my case), and the results will be different. If you changed the number of simulations (bootstrap samples), 1e4 in my case, the results wont change that much, especially if you increase it, because this is a relatively easy task and can be well approximated without many simulations. $\endgroup$ Oct 26, 2020 at 11:07

I think one possible solution for this test is to

  1. turn to regression to get the two means
  2. calculate the absolute value of their difference from the regression coefficients (a non-linear combination). Let's call this random variable $|\Delta|$.

Once this is done, you have two choices.

You can look at the one-sided CI for $|\Delta|$ to see if it excludes your superiority threshold. You can get that easily from step (2), since the overlap between two one-sided 95% CIs makes a two-sided 90% CI, so you can work backwards from the usual 90% CI for $|\Delta|$.

Alternatively, you can perform a two-sided hypothesis test on $|\Delta|$, and then calculate the one-sided p-value from that. This is a bit more work, but is just a matter of getting the sign of the inequality, a $\chi^2$ statistic from the two-sided test, and evaluating cumulative standard normal distribution. If your test returns an F-statistic, you will have to use that instead, along with the t distribution in place of the normal. If you don't want to go this route, when $|\Delta| - \delta$ is positive, you can simply divide the two-sided p-value by 2. In the other case, you need to calculate $1-\frac{p}{2}$ since you are in the other tail. This simpler division approach works for symmetric distributions only.

Here is an example in Stata, where we will conduct two such hypotheses comparing the average price of foreign (foreign = 1) and domestic cars (foreign = 0):

  . sysuse auto, clear
(1978 Automobile Data)

. table foreign, c(mean price)

 Car type | mean(price)
 Domestic |     6,072.4
  Foreign |     6,384.7

. /* (1) Calculate the means using regression */
. regress price ibn.foreign, noconstant

      Source |       SS           df       MS      Number of obs   =        74
-------------+----------------------------------   F(2, 72)        =    159.91
       Model |  2.8143e+09         2  1.4071e+09   Prob > F        =    0.0000
    Residual |   633558013        72  8799416.85   R-squared       =    0.8162
-------------+----------------------------------   Adj R-squared   =    0.8111
       Total |  3.4478e+09        74  46592355.7   Root MSE        =    2966.4

       price |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
     foreign |
   Domestic  |   6072.423    411.363    14.76   0.000     5252.386     6892.46
    Foreign  |   6384.682   632.4346    10.10   0.000     5123.947    7645.417

. /* (2) Calculate the absolute value of the foreign-domestic difference */
. nlcom av_diff:abs(_b[1.foreign] - _b[0.foreign]), level(90) post

     av_diff:  abs(_b[1.foreign] - _b[0.foreign])

       price |      Coef.   Std. Err.      z    P>|z|     [90% Conf. Interval]
     av_diff |   312.2587   754.4488     0.41   0.679    -928.6992    1553.217

. /* (3a) We know that a one-sided 95% CI is (-inf,1553.217] */
. /* (3b) Transform two-sided test into a one-sided test and get p-values */
. // Test something just inside the CI */
. // H_0': (avg_price_foreign - avg_price_domestic) <= 1553
. // H_1': (avg_price_foreign - avg_price_domestic)  > 1553
. test av_diff = 1553

 ( 1)  av_diff = 1553

           chi2(  1) =    2.70
         Prob > chi2 =    0.1001

. local sign_av_diff = sign(_b[av_diff] - 1553) // get the sign

. display "p-value' = " normal(`sign_av_diff'*sqrt(r(chi2)))
p-value' = .05002962

. // Test something just above the CI */
. // H_0'': (avg_price_foreign - avg_price_domestic) <= 1554
. // H_1'': (avg_price_foreign - avg_price_domestic)  > 1554
. test av_diff = 1554

 ( 1)  av_diff = 1554

           chi2(  1) =    2.71
         Prob > chi2 =    0.0998

. local sign_av_diff = sign(_b[av_diff] - 1554) // get the sign

. display "p-value = " normal(`sign_av_diff'*sqrt(r(chi2)))
p-value = .049893

The one-sided 95% CI is $(-\infty, 1553.217]$, so $\delta>1553.217$ in order of us to reject. If we try testing a value below that upper bound like 1553, the one-sided p-value is .05003, so we cannot reject. If we test something just above the UB, like 1554, the p-value is .049893, so we can reject at $\alpha=5\%$. I don't advocate using rigid thresholds for significance, this is just meant to illustrate the intuition. Note that you can also divide the two-sided p-values by 2 to get this (Stata's two-sided p-values are on the "Prob > chi2" line).

Here the null is $H_0=|\Delta|\le \delta$ (practical equivalence) versus $H_a=|\Delta| > \delta$ (non-equivalence). We focus on testing $|\Delta| = \delta$, so we calculate the probability at the most extreme point of the null hypothesis, closest to alternative parameter space. This means that the p-value is exact only for $|\Delta| = \delta$. If $|\Delta| < \delta$, then our p-value is just a conservative bound on the type I error rate (the error being finding a negative effect when there is none).

  • $\begingroup$ Thanks for your answer. Is this a code for R? How do I load the sample data? $\endgroup$ Oct 26, 2020 at 10:21
  • 1
    $\begingroup$ That is stata code. $\endgroup$ Oct 26, 2020 at 12:02
  • 1
    $\begingroup$ @BlueVarious I think you should be able to do this in R. I expanded on the approach a bit to make it clearer what I was doing. If something is still unclear, I am happy to clarify. You can get then Stata dataset here. $\endgroup$
    – dimitriy
    Oct 26, 2020 at 18:30

You can perform a t-test and just look at confidence intervals. In some circumstances (e.g. clinical trials) you are not interested in statistical significance, but whether the difference is significant from a practical point of view by adding a margin $\delta$ (in a clinical trials setting it’s called clinical significance). Have a look at the picture. We assess mean response difference in experimental and control group. enter image description here

  • 2
    $\begingroup$ This image is about the null hypothesis 'that there is no equivalence'. The OP is asking the question for the null hypothesis 'that there is equivalence'. $\endgroup$ Oct 26, 2020 at 13:26
  • $\begingroup$ @Sextus Empiricus If it becomes significant under my null hypothesis, ( ■ ) will be either to the right of the δ line or to the left of the -δ line on the treskov's figure, am I right? $\endgroup$ Oct 27, 2020 at 10:43
  • 1
    $\begingroup$ @BlueVarious that is more or less right. It would be extending the idea of a t-test for a point $\mu_A-\mu_B=0$ to a region by just shifting the boundaries. But I believe there can be made some improvement. $\endgroup$ Oct 27, 2020 at 10:55

one of ideas is to add $\delta$ to one population (raising mean) and in second test substracting $\delta$ and then computing statistic and figure out in two "one-sided tests" p-values, after adding these you will have one p-value for two sided test stated in your question

it's like solving equation in elementary school:

$$|\mu_A - \mu_B| \le \delta => \begin{cases} \mu_A - \mu_B \le \delta, & \text{if}\ \mu_A - \mu_B \ge 0 \\[2ex] \mu_A - \mu_B \ge -\delta, & \text{if}\ \mu_A - \mu_B < 0 \end{cases} =>\begin{cases} (\mu_A-\delta) - \mu_B \le 0, & \text{if}\ \mu_A - \mu_B \ge 0 \\[2ex] (\mu_A+\delta) - \mu_B \ge 0, & \text{if}\ \mu_A - \mu_B < 0 \end{cases} =>\begin{cases} (\mu_A-\delta) \le \mu_B, & \text{if}\ \mu_B \le \mu_A\\[2ex] (\mu_A+\delta) \ge \mu_B, & \text{if}\ \mu_B > \mu_A \end{cases}$$ this is your $H_0$ :) now let's construct $H_1$ $$H_0\begin{cases} (\mu_A-\delta) \le \mu_B, & \text{if}\ \mu_B \le \mu_A\\[2ex] (\mu_A+\delta) \ge \mu_B, & \text{if}\ \mu_B > \mu_A \end{cases}, H_1\begin{cases} (\mu_A-\delta) \ge \mu_B, & \text{if}\ \mu_B \le \mu_A, & (1)\\[2ex] (\mu_A+\delta) \le \mu_B, & \text{if}\ \mu_B > \mu_A, & (2) \end{cases}$$

for $(1)$ you want to compute p-value that $$p((\mu_A-\delta) \ge \mu_B|\mu_A \ge \mu_B) = \frac{p((\mu_A-\delta) \ge \mu_B)}{p(\mu_A \ge \mu_B)}$$ analogous for $(2)$,

and combining $$p(|\mu_A - \mu_B| \le \delta) = 1-p((\mu_A-\delta) \ge \mu_B|\mu_A \ge \mu_B) - p((\mu_A+\delta) \le \mu_B|\mu_A \lt \mu_B)$$

ask questions if needed, I am not entirely sure of this approach, and would welcome any critique

  • $\begingroup$ Thank you for your answer. How is it calculated using the question statement $\tau_{\phi ,\mu}$ ? $\endgroup$ Oct 26, 2020 at 10:19
  • 1
    $\begingroup$ Can you give a link or reference for this? $\endgroup$ Oct 26, 2020 at 12:01
  • $\begingroup$ @quester Thanks for adding the explanation. What you are calculating appears to be $p ({H}_{1}\ is\ true )$, am I right? If so, how does how to calculate the p-value from $p ({H}_{1}\ is\ true )$?daniellakens.blogspot.com/2015/11/… Furthermore, there is the question of what is the probability distribution of the $p()$? $\endgroup$ Oct 27, 2020 at 9:59
  • $\begingroup$ I want to share the premise of the discussion. I understand the p-value is the probability of an event occurring more abnormal than the observed value, under the assuming $H_0$ is true, am I right? If so, the p-value might be $p(|T|>|t_{obs}|,{H_0}\ is\ true)$ under my notations. Here, $T$ is the random variable representing the t-value. $\endgroup$ Oct 27, 2020 at 10:33

Just for a comment;

Let $F$ be the cumulative distribution of $p(\ \ |{H_0})$, that means

$$F(t) = p(t>T |\ {H_0}\ is\ true) . \tag{1-1}$$

Here, $p(t>-\infty\ |\ {H_0}\ is\ true)$ is the probability that $t>T$ under the condition that $H_0$ is true, $T$ is a random value representing the t-value. The $t$ is a real number substituted to the $F$.

And, let $t_{obs}$ be the t-value calculated from actual observations.

Then, the p-value shall be; $$p-value = p(|t|>|t_{obs}|\ |\ {H_0}\ is\ true). \tag{1-2}$$

Therefore, $$p-value = p(|t|>|t_{obs}|\ |\ {H_0}\ is\ true)$$ $$=p(\ t>|t_{obs}|\ or\ \ t<-|t_{obs}|\ |\ {H_0}\ is\ true)$$ $$=p(\ t>|t_{obs}|\ |\ {H_0}\ is\ true)\ +\ p(\ t<-|t_{obs}|\ |\ {H_0}\ is\ true) $$ $$=F(-|t_{obs}|)+(1-F(|t_{obs}|))$$ $$=1+F(-|t_{obs}|)-F(|t_{obs}|) \tag{1-3}$$

Thus, the essence of my question would be what function $F$ in (1-1) would be under my ${H}_{0}$.

If the mean and standard deviation of the population are known, I think these distributions can be brought to a form similar to the simulation of user2974951 by using the regenerability of the normal distribution.

However, if both of the mean and standard deviation of the population are unknown, then I have no idea.

I'm waiting for your opinion.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.