An investigator wishes to produce a combined analysis of several datasets. In some datasets there are paired observations for treatment A and B. In others there are unpaired A and/or B data. I am looking for a reference for an adaptation of the t-test, or for a likelihood ratio test, for such partially paired data. I am willing to (for now) to assume normality with equal variance and that the population means for A are the same for each study (and likewise for B).

  • 2
    $\begingroup$ Hi Frank. Maybe it would be helpful to make the modeling assumptions more explicit. Normally when I think of paired designs, I think of one of the following (i) trying to remove fixed unobservable unit-level effects, (ii) reducing the variability of a random effect across experimental units, or (iii) adjusting for the nonnormality of the response by taking differences between the pairs, thus obtaining a better approximation. In particular, I do not immediately see any benefit in matched pairs if the assumption under the null is that the observations are all iid normal. $\endgroup$
    – cardinal
    Commented Apr 6, 2012 at 3:07
  • 4
    $\begingroup$ Cardinal, I've actually got a lot of data that looks like this too. We were trying to collect fully paired data, but due to technical problems or bad luck, some samples measurements under A or B are sometimes spoiled. The two obvious--but unsatisfying--solutions are to 1) throw out all incomplete pairs and do a paired t-test, or 2) ignore the pairing and do an unpaired t-test across all the data. I think the poster is asking for a way to leverage the pairing where it exists (for your reason #1 and #2), while salvaging whatever he can from the other, unpaired, data points. $\endgroup$ Commented Apr 6, 2012 at 7:57
  • 2
    $\begingroup$ I appreciate all the comments. For the matched pairs subjects were tested under both A and B. One way to leverage the pairing is to use the bootstrap nonparametric percentile confidence interval for the difference between means in A and B. This would involve using the cluster bootstrap, sampling with replacement from subjects. A subject that doesn't have paired data would have one observation kept or deleted in a resample, and paired data would have two records kept or deleted. This seems to respect the pairing but an estimand needs defining and we don't know about optimality. $\endgroup$ Commented Apr 6, 2012 at 12:34
  • 1
    $\begingroup$ The Bayesian approach is easy to implement. $\endgroup$ Commented Jun 5, 2012 at 20:28
  • 2
    $\begingroup$ Hani M. Samawi & Robert Vogel , Journal of Applied Statistics (2013): Notes on two sample tests for partially correlated (paired) data, dx.doi.org/10.1080/02664763.2013.830285 $\endgroup$
    – Suresh
    Commented Oct 25, 2013 at 13:21

7 Answers 7


Guo and Yuan suggest an alternative method called the optimal pooled t-test stemming from Samawi and Vogel's pooled t-test.


Guo B, Yuan Y. A comparative review of methods for comparing means using partially paired data. Statistical Methods in Medical Research. 2017;26(3):1323-1340. doi:10.1177/0962280215577111. Open access link: https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=ef2b294855f1acadbe5b04cfd86f61b1275597cc

Great read with multiple options for this situation.

New to commenting so please let me know if I need to add anything else.


Well, if you knew the variances in the unpaired and in the paired (which would generally be a good deal smaller), the optimal weights for the two estimates of difference in groups means would be to have weights inversely proportional to the variance of the individual estimates of the difference in means.

[Edit: turns out that when the variances are estimated, this is called the Graybill-Deal estimator. There's been quite a few papers on it. Here is one]

The need to estimate variance causes some difficulty (the resulting ratio of variance estimates is F, and I think the resulting weights have a beta distribution, and a resulting statistic is kind of complicated), but since you're considering bootstrapping, this may be less of a concern.

An alternative possibility which might be nicer in some sense (or at least a little more robust to non-normality, since we're playing with variance ratios) with very little loss in efficiency at the normal is to base a combined estimate of shift off paired and unpaired rank tests - in each case a kind of Hodges-Lehmann estimate, in the unpaired case based on medians of pairwise cross-sample differences and in the paired case off medians of pairwise-averages-of-pair-differences. Again, the minimum variance weighted linear combination of the two would be with weights proportional to inverses of variances. In that case I'd probably lean toward a permutation (/randomization) rather than a bootstrap - but depending on how you implement your bootstrap they can end up in the same place.

In either case you might want to robustify your variances/shrink your variance ratio. Getting in the right ballpark for the weight is good, but you'll lose very little efficiency at the normal by making it slightly robust. ---

Some additional thoughts I didn't have clearly enough sorted out in my head before:

This problem has distinct similarities to the Behrens-Fisher problem, but is even harder.

If we fixed the weights, we could just whack in a Welch-Satterthwaite type approximation; the structure of the problem is the same.

Our issue is that we want to optimize the weights, which effectively means weighting is not fixed - and indeed, tends to maximize the statistic (at least approximately and more nearly in large samples, since any set of weights is a random quantity estimating the same numerator, and we're trying to minimize the denominator; the two aren't independent).

This would, I expect, make the chi-square approximation worse, and would almost surely affect the d.f. of an approximation still further.

[If this problem is doable, there also just might turn out be a good rule of thumb that would say 'you can do almost as well if you use only the paired data under these sets of circumstances, only the unpaired under these other sets of conditions and in the rest, this fixed weight-scheme is usually very close to optimal' -- but I won't hold my breath waiting on that chance. Such a decision rule would doubtless have some impact on true significance in each case, but if that effect wasn't so big, such a rule of thumb would give an easy way for people to use existing legacy software, so it could be desirable to try to identify a rule like that for users in such a situation.]


Edit: Note to self - Need to come back and fill in details of work on 'overlapping samples' tests, especially overlapping samples t-tests


It occurs to me that a randomization test should work okay -

  • where the data are paired you randomly permute the group labels within pairs

  • where data are unpaired but assumed to have common distribution (under the null), you permute the group assignments

  • you can now base weights to the two shift estimates off the relative variance estimates ($w_1 = 1/(1+\frac{v_1}{v_2})$), compute each randomized sample's weighted estimate of shift and see where the sample fits into the randomization distribution.

(Added much later)

Possibly relevant paper:

Derrick, B., Russ B., Toher, D., and White, P. (2017),
"Test Statistics for the Comparison of Means for Two Samples That Include Both Paired and Independent Observations"
Journal of Modern Applied Statistical Methods, May, Vol. 16, No. 1, 137-157.
doi: 10.22237/jmasm/1493597280

  • 1
    $\begingroup$ +1. I have a question about the last part of your answer. What variance estimates (i.e. what weights) would you use in the permutation test - the actual ones computed on the actual sample, or would you compute the weights for each permutation based on the data from that permutation? $\endgroup$
    – amoeba
    Commented May 21, 2017 at 21:35
  • $\begingroup$ @amoeba to properly account for the sample-based nature of the calculation you'd base it on that particular permutation. $\endgroup$
    – Glen_b
    Commented May 25, 2017 at 4:22
  • $\begingroup$ @amoeba I should compare that with some other approaches to the problem. $\endgroup$
    – Glen_b
    Commented May 29, 2017 at 14:10
  • 1
    $\begingroup$ By the way, I came across this thread because somebody approached me with the following data: two subjects with paired data plus two subjects with unpaired data (i.e. 3 measurements in group A, 3 measurements in group B, out of these 6 values 2+2 are paired and the rest is unpaired). In this case it's not enough data to estimate the variance of the unpaired shift estimate, so I could not suggest anything apart from ignoring the pairing and doing an unpaired test... But this is of course quite an extreme situation. $\endgroup$
    – amoeba
    Commented May 29, 2017 at 16:13

Here are some thoughts. I basically just arrive to Greg Snow conclusion that This problem has distinct similarities to the Behrens-Fisher problem. To avoid handwaving I first introduce some notations and formalize the hypotheses.

  • we have $n$ paired observations $x_i^{pA}$ and $x_i^{pB}$ ($i = 1, \dots, n$);
  • we have $n_A$ and $n_B$ unpaired observations $x_i^A$ ($i = 1, \dots, n_A$) and $x_i^B$ ($i = 1, \dots, n_B$);
  • each observation is the sum of a patient effect and a treatment effect. The corresponding random variables are

    • $X_i^{pA} = P_i + T_i^A$, $X_i^{pB} = P_i + T_i^B$,
    • $X_i^A = Q_i + U_i^A$, $X_i^B = R_i + V_i^B$

    with $P_i, Q_i, R_i \sim \mathcal N(0,\sigma_P^2)$, and $T_i^\tau, U_i^\tau, V_i^\tau \sim \mathcal N(\mu_\tau, \sigma^2)$ ($\tau = A, B$).

    • under the null hypothesis, $\mu_A = \mu_B$.

We form as usual a new variable $X_i = X_i^{pA} - X_i^{pB}$. We have $X_i \sim \mathcal N(\mu_A - \mu_B, 2\sigma^2)$.

Now we have three groups of observations, the $X_i$ (size $n$), the $X_i^A$ (size $n_A$) and the $X_i^B$ (size $n_B$). The means are

  • $X_\bullet\sim \mathcal N(\mu_A - \mu_B, {2\over n} \sigma^2)$
  • $X^A_\bullet\sim \mathcal N(\mu_A , {1\over n_A} (\sigma_P^2 + \sigma^2))$
  • $X^B_\bullet\sim \mathcal N(\mu_B , {1\over n_B} (\sigma_P^2 + \sigma^2))$

The next natural step is to consider

  • $Y = X_\bullet + X^A_\bullet - X^B_\bullet \sim \mathcal N\left( 2(\mu_A-\mu_B), {2\over n} \sigma^2 + \left({1\over n_A}+ {1\over n_B}\right) (\sigma_P^2 + \sigma^2)\right)$

Now basically we are stuck. The three sums of squares give estimations of $\sigma^2$ with $n-1$ df, $\sigma_P^2 + \sigma^2$ with $n_A-1$ df and $n_B-1$ df respectively. The last two can be combined to give an estimation of $\left({1\over n_A}+ {1\over n_B}\right) (\sigma_P^2 + \sigma^2)$ with $n_A+n_B-2$ df. The variance of $Y$ is the sum of two terms, each of which can be estimated, but the recombination is not doable, just as in Behrens Fisher problem.

At this point I think one may plug-in any solution proposed to Behrens Fisher problem to get a solution to your problem.

  • 1
    $\begingroup$ I fixed a few typos in formulas. Please check! $\endgroup$ Commented Jun 12, 2015 at 21:27

My first thought was a mixed effects model, but that has already been discussed so I won't say any more on that.

My other thought is that if it were theoretically possible that you could have measured paired data on all subjects but due to cost, errors, or another reason you don't have all the pairs, then you could treat the unmeasured effect for the unpaired subjects as missing data and use tools like the EM algorithm or Multiple Imputation (missing at random seems reasonable unless the reason a subject was only measured under 1 treatment was related to what their outcome would be under the other treatment).

It may be even simpler to just fit a bivariate normal to the data using maximum likelihood (with the likelihood factored based on the available data per subject), then do a likelihood ratio test comparing the distribution with the means equal vs. different means.

It has been a long time since my theory classes, so I don't know how these compare on optimality.

  • 1
    $\begingroup$ Thanks Greg. I'm leaning towards the customized maximum likelihood approach. $\endgroup$ Commented Jun 5, 2012 at 18:12

maybe mixed modelling with patient as random effect could be a way. With mixed modelling the correlation structure in the paired case and the partial missings in the unpaired case could be accounted for.

  • 2
    $\begingroup$ Since none of the datasets when analyzed individually would lead one to use random effects, I'm not seeing why random effects are helpful here. But it may be possible to use generalized least squares to allow each subject to have her own correlation structure. Unpaired observations would have correlation zero. Worth thinking about. Thanks. $\endgroup$ Commented Apr 6, 2012 at 3:04
  • $\begingroup$ yes, you're right, the datasets would not require mixed modelling if used separately. But if you append them into one single dataset, you could use the approach to incorporate the correlation in the paired data and simultaneously use the unpaired data by specifying a zero correlation. $\endgroup$
    – psj
    Commented Apr 7, 2012 at 13:17
  • 1
    $\begingroup$ Yes; my point was that a mixed model may be an overkill as you can easily specify the subject-varying correlation structure using generalized least squares (using e.g. R's gls function in the nlme4 package. $\endgroup$ Commented Apr 7, 2012 at 15:51
  • $\begingroup$ A new approach using GEE or random effects with an ordinal regression is simple. See hbiostat.org/bbr/nonpar. This is based on generalizing the rank difference test. $\endgroup$ Commented Jan 17 at 13:37

One of the methods proposed in Hani M. Samawi & Robert Vogel (Journal of Applied Statistics , 2013) consists of a weighted combination of T-scores from independent and dependent samples in such a way that the new T score is equal to

$T_o = \sqrt\gamma ( \frac {\mu_Y - \mu_X} {S_x^2/n_X + S_y^2/n_Y}) + \sqrt {(1-\gamma)} \frac {\mu_D} {S_D^2/n_D}$

where $D$ represents the samples of paired differences taken from the correlated data. Basically the new T score is a weighted combination of the unpaired T-score with the new correction term. $\gamma$ represents the proportion of independent samples. When $\gamma$ is equal to 1 the test is equivalent to two sample t-test, whereas if equal to zero, it is a paired t-test.

  • 2
    $\begingroup$ Regarding this conservative weighted test for partially correlated data, the authors seem to distinguish several cases (large sample approximation, unequal ratio of matched/independent samples, small samples) in order to get the distribution of $T_0$ under the null. It would be helpful if you could summarize them in your answer, since the same article is referenced in another (partial) reply. $\endgroup$
    – chl
    Commented May 21, 2014 at 11:50

Building on Frank's comment on Apr 7, 2012 at 15:51, about "gls", below is a small example using gls from R package nlme. As correlation structure I used heterogeneous compound symmetry. There are two incomplete "people" with data on only one occasion.

occasion <- factor(rep(1:2,10))
y1 <- c(1,2,3,4,NA,6,7,8,9,10)
y2 <- c(3,4,6,2,5,8,7,NA,9,11)
id <- rep(1:10,each=2)
da <- data.frame(id,occasion, y=c(y1,y2))
da <- da[complete.cases(da),]

HCS.gls <- gls (y ~ 1+occasion, data=da,
                correlation = corSymm(form = ~ occasion|id ),
                weights = varIdent (form = ~1 | occasion) ) 

The results are as follows:

Correlation Structure: Compound symmetry
 Formula: ~occasion | id 
 Parameter estimate(s):
Variance function:
 Structure: Different standard deviations per stratum
 Formula: ~1 | occasion 
 Parameter estimates:
       1        2 
1.000000 1.209334 

               Value Std.Error  t-value p-value
(Intercept) 5.536308 0.8678529 6.379316  0.0000
occasion2   0.729696 0.6964471 1.047740  0.3103

occasion2 -0.113

Standardized residuals:
        Min          Q1         Med          Q3         Max 
-1.68771470 -0.88199849  0.04533997  0.54456058  1.45639118 

Residual standard error: 2.687841 
Degrees of freedom: 18 total; 16 residual

The residual standard error 2.687841 at the bottom shows the std. deviation of y at occasion 1. For occasion 2, this value must be multiplied by 1.209334 as is shown under "Parameter estimate(s)". Under "Coefficients" the mean of y at occasion 1 is shown to be 5.536308. At occasion 2 the mean is 0.729696 higher, and the p value associated with this difference between the two occasions is 0.3103.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.