I have data from two populations (let's call them "before" and "after"), collected in a paired fashion. That is, I take a measurement under one set of conditions before, and then I can take multiple measurements under the same set of conditions after. For example, the "pairs" in this case might consist of $(X^{b,i}, (X^{a,i}_{1}, X^{a,i}_{2}))$, where $X^{b,i}$ is the measurement from the "before" population taken under conditions i, and $X^{a,i}_1$ and $X^{a,i}_2$ are two measurements from the "after" population taken under conditions i. I have this data for $i=1, \ldots, n$ sets of conditions.

I want to test for equivalence of the means of the two populations. A paired two one-sided t-tests approach seems appropriate, and I've been using the implementation available in the statsmodels Python package.

How should I handle the unequal sample sizes when performing a paired two one-sided t-tests?

My current approach is to turn each $(X^{b,i}, (X^{a,i}_{1}, X^{a,i}_{2}))$ pair into two pairs: $(X^{b,i}, X^{a,i}_{1})$ and $(X^{b,i}, X^{a,i}_{2})$. Then I use a paired two one-sided t-tests on this new larger set of pairs (with $2n$ new pairs). Is this appropriate? I worry that the new pairs are not independent (because they share $X^{b,i}$), and that this will invalidate this approach.


3 Answers 3


I don't know any specific references for this case.

In analogy to some of the methods for repeated measures ANOVA, the relevant t-test would use the mean of the two 'after' observations and compare it with the 'before' observation. The variance of the average within difference will be smaller with more observations per individual, so the test still takes the larger sample into account.

An alternative approach would be to use cluster robust standard errors for the two pair differences as in your approach. This would be possible with OLS with the two pairs sample and specify the individual as grouping variable. OLS in statsmodels only has t-test after estimation, but the TOST rejection decision could still be obtained by comparing the 2 alpha confidence interval with the equivalence boundaries.

about cluster robust standard errors:

OLS provides a consistent estimator of the parameters for the linear model even if there is correlation across observations or heteroscedasticity. However, the usual estimate for the standard errors or covariance of the parameter estimates is incorrect. One possible solution to correlation and heteroscedasticity is to use the OLS parameter estimates, but correct the standard errors by using a sandwich form of robust standard errors.

For example here is the Wikipedia page for heteroscedasticity robust standard errors http://en.wikipedia.org/wiki/Heteroscedasticity-consistent_standard_errors

For the specific case when we have correlation within small groups or clusters but no correlation across groups, we can use cluster robust standard errors to correct for the within cluster correlation. An extensive discussion is available in

Cameron, A. Colin, and Douglas L. Miller. "A practitioner’s guide to cluster-robust inference."

(aside: statsmodels provides robust covariance matrices for the linear model, OLS, WLS, for discrete models like Logit and Poisson, for GLM. Cluster robust standard errors are the default for GEE. The list of provided types is here http://statsmodels.sourceforge.net/devel/generated/statsmodels.regression.linear_model.RegressionResults.get_robustcov_results.html )

  • $\begingroup$ Thanks for these thoughts! I see how using the mean of the two 'after' observations makes sense, and I might just do that. Can you please point me to a reference that describes "cluster robust standard errors" and their connection with OLS? $\endgroup$
    – mbloem
    Commented Jun 3, 2015 at 22:36
  • $\begingroup$ I added some information and links for robust standard errors to my answer. $\endgroup$
    – Josef
    Commented Jun 3, 2015 at 23:01

Normally with repeated outcome variables one would use a mixed effects ANOVA. The baseline assessment is considered fixed, or given, so it's simply a matter of using an offset. The inference is then based around the intercept, $\beta_0$ of the following linear model:

\begin{equation} X^{b,i} - \mbox{offset}(X^{a, i}) = \beta_0 + \gamma_i \end{equation}

where $\gamma_i$ is the normally distributed, mean 0 random effect accounting for the intracluster variability of the outcome.

  • $\begingroup$ The after variables are a matrix of measures on one population. The before population is independent. You are not using 'pairs' in the way statisticians typically do. $\endgroup$
    – Georgette
    Commented Jul 19, 2018 at 19:57

There's some software here that will compute this for you: http://clintools.com/products/freeware.html - I wrote it, so if you see anything not to your liking let me know.

That said, I don't understand how you can have unequal numbers in a paired samples t-test or TOST. Basically, if one drops out, the pair must also.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.