I'm looking at Lakens' (2017) primer and he tests the hypothesis that there is a difference between two groups $X_1$and $X_2$of magnitude $\Delta=E[X_1]-E[X_2] \neq0$ by subtracting $\Delta$ from the difference between the sample means $d=M_1-M_2$ and then he computes the $t$ value and $p$ value based on this difference score with a procedure analogous to Welch t-test (Eqs 3 and 4 on p. 357).

This looks wrong to me. IMO to determine the p value, one should be using non-central $t$ distribution with non-centrality parameter based on $\Delta$ and t value based on $d$. This is because under the hypothesis, $d$ and hence the test statistic $t=d/h$ (where we would substitute $h=\sqrt{s_1^2/n_1+s_2^2/n_2}$ following Welch) does not have a symmetric distribution because $E[X_1]-E[X_2]=\Delta\neq 0$. Subtracting $\Delta$ from $d$ does merely shift the distribution, but it can't make it symmetric, since $\Delta$ is not a random variable.

Another CrossValidated answer claims that

Using the zero centered T distribution here would test the hypothesis assuming that X1 - X2 -3 is symmetric about zero.`

(X1-X2-3 corresponds to $X_1-X_2-\Delta$). I don't see how it is possible to obtain such symmetric test statistic distribution unless $E[X_1]=E[X_2]$ which under the hypothesis is not the case ($E[X_1]-E[X_2]=\Delta\neq 0$).

So, which is the correct procedure to conduct equivalence test?


Lakens, D. (2017). Equivalence tests: a practical primer for t tests, correlations, and meta-analyses. Social Psychological and Personality Science, 8(4), 355-362.


What you are proposing does not appear to be any different from what actually occurs in this test. Remember that in a classical hypothesis test, the p-value is calculated using the null distribution of the test statistic. This calculation assumes that the null hypothesis is true. You suggest that the calculation of the p-value should use the non-central T-distribution with non-centrality parameter $\Delta$. However, the null hypothesis for this test is that $\Delta = 0$, so under that condition the non-central T-distribution simplifies down to the standard (centralised) T-distribution.

| cite | improve this answer | |

You are correct, in the sense that the uniformly most powerful (UMP) test for equivalence for the one-sample, paired, and two-sample $T$-test does involve a non-central $t$-distribution.

The two one-sided tests (TOST) procedure with both rejection regions with size $\alpha$, or equivalently the $1 - 2 \alpha$ confidence-interval inclusion procedure, has size $\alpha$ when the null hypothesis is true. However, it is not as powerful as the UMP test for small sample sizes. One can show that, as the sample size grows, the TOST procedure is asymptotically equivalent to the UMP test. But a great deal of power can be lost for small sample sizes.

This code simulates from a standard Normal distribution and performs the test for equivalence in the one-sample setting using both the TOST test (via Daniel Lakens TOSTER package) and the UMP test, where the P-value for the UMP test is given by $$P(|T| < |t_{\text{obs}}|; \texttt{df} = n - 1, \texttt{ncp} = \sqrt{n}\cdot\delta/\sigma)$$ where $T$ is a non-central $t$-random variable and $t_{\text{obs}} = \frac{\bar{x}}{s/\sqrt{n}}$.

n <- 10
alpha <- 0.05

theta <- 0.5 # delta / sigma

ncp <- theta*sqrt(n) # delta/sigma*sqrt(n)

# Simulate 10000 samples, and compute power.
S <- 10000

# theta.true <- theta # Null is true
theta.true <- 0 # Alternative is true

Pvals.tost <- rep(NA, length(S))
Pvals.ump  <- rep(NA, length(S))

for (s in 1:S){
  Xs <- rnorm(n, mean = theta.true)

  xbar <- mean(Xs)

  tobs <- xbar/(sd(Xs)/sqrt(n))

  Pvals.ump[s] <- pt(abs(tobs), df = n-1, ncp = ncp) - pt(-abs(tobs), df = n-1, ncp = ncp)

  out <- TOSTER::TOSTone(m = xbar,
                  mu = 0,
                  sd = 1,
                  n = n,
                  low_eqbound_d = -theta,
                  high_eqbound_d = theta,
                  alpha = 0.05,
                  plot = FALSE,
                  verbose = FALSE)

  Pvals.tost[s] <- max(out$TOST_p1, out$TOST_p2)

mean(Pvals.tost <= 0.05)
mean(Pvals.ump <= 0.05)

With $n = 10$, the TOST procedure has 0 power to detect the discrepancy from the null, while the UMP test has power ~ 0.17 to detect it.

Reference: Stefan Wellek, Testing Statistical Hypotheses of Equivalence and Noninferiority, pages 64 and 92.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.