Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

An investigator wishes to produce a combined analysis of several datasets. In some datasets there are paired observations for treatment A and B. In others there are unpaired A and/or B data. I am looking for a reference for an adaptation of the t-test, or for a likelihood ratio test, for such partially paired data. I am willing to (for now) to assume normality with equal variance and that the population means for A are the same for each study (and likewise for B).

share|improve this question
Well, the test statistics for each data set separately all have an (asymptotic) $N(0,1)$ distribution. Since the data set are presumably independent, why not take the average and treat it as a usual $z$-statistic? – Macro Apr 5 '12 at 22:09
I'm not clear on what you are suggesting. And I don't want to combine test statistics. I want to have an optimal test. – Frank Harrell Apr 6 '12 at 3:02
Hi Frank. Maybe it would be helpful to make the modeling assumptions more explicit. Normally when I think of paired designs, I think of one of the following (i) trying to remove fixed unobservable unit-level effects, (ii) reducing the variability of a random effect across experimental units, or (iii) adjusting for the nonnormality of the response by taking differences between the pairs, thus obtaining a better approximation. In particular, I do not immediately see any benefit in matched pairs if the assumption under the null is that the observations are all iid normal. – cardinal Apr 6 '12 at 3:07
Cardinal, I've actually got a lot of data that looks like this too. We were trying to collect fully paired data, but due to technical problems or bad luck, some samples measurements under A or B are sometimes spoiled. The two obvious--but unsatisfying--solutions are to 1) throw out all incomplete pairs and do a paired t-test, or 2) ignore the pairing and do an unpaired t-test across all the data. I think the poster is asking for a way to leverage the pairing where it exists (for your reason #1 and #2), while salvaging whatever he can from the other, unpaired, data points. – Matt Krause Apr 6 '12 at 7:57
I appreciate all the comments. For the matched pairs subjects were tested under both A and B. One way to leverage the pairing is to use the bootstrap nonparametric percentile confidence interval for the difference between means in A and B. This would involve using the cluster bootstrap, sampling with replacement from subjects. A subject that doesn't have paired data would have one observation kept or deleted in a resample, and paired data would have two records kept or deleted. This seems to respect the pairing but an estimand needs defining and we don't know about optimality. – Frank Harrell Apr 6 '12 at 12:34

This is a problem that comes up so much I am kind of surprised that I haven't seen it explicitly tackled.

Well, if you knew the variances in the unpaired and in the paired (which would generally be a good deal smaller), the optimal weights for the two estimates of difference in groups means would be to have weights inversely proportional to the variance of the individual estimates of the difference in means.

The need to estimate variance causes some difficulty (the resulting ratio of variance estimates is F, and I think the resulting weights have a beta distribution, and a resulting statistic is kind of complicated), but since you're considering bootstrapping, this may be less of a concern.

An alternative possibility which might be nicer in some sense (or at least a little more robust to non-normality, since we're playing with variance ratios) with very little loss in efficiency at the normal is to base a combined estimate of shift off paired and unpaired rank tests - in each case a kind of Hodges-Lehmann estimate, in the unpaired case based on medians of pairwise cross-sample differences and in the paired case off medians of pairwise-averages-of-pair-differences. Again, the minimum variance weighted linear combination of the two would be with weights proportional to inverses of variances. In that case I'd probably lean toward a permutation (/randomization) rather than a bootstrap - but depending on how you implement your bootstrap they can end up in the same place.

In either case you might want to robustify your variances/shrink your variance ratio. Getting in the right ballpark for the weight is good, but you'll lose very little efficiency at the normal by making it slightly robust. ---

Some additional thoughts I didn't have clearly enough sorted out in my head before:

This problem has distinct similarities to the Behrens-Fisher problem, but is even harder.

If we fixed the weights, we could just whack in a Welch-Satterthwaite type approximation; the structure of the problem is the same.

Our issue is that we want to optimize the weights, which effectively means weighting is not fixed - and indeed, tends to maximize the statistic (at least approximately and more nearly in large samples, since any set of weights is a random quantity estimating the same numerator, and we're trying to minimize the denominator; the two aren't independent).

This would, I expect, make the chi-square approximation worse, and would almost surely affect the d.f. of an approximation still further.

[If this problem is doable, there also just might turn out be a good rule of thumb that would say 'you can do almost as well if you use only the paired data under these sets of circumstances, only the unpaired under these other sets of conditions and in the rest, this fixed weight-scheme is usually very close to optimal' -- but I won't hold my breath waiting on that chance. Such a decision rule would doubtless have some impact on true significance in each case, but if that effect wasn't so big, such a rule of thumb would give an easy way for people to use existing legacy software, so it could be desirable to try to identify a rule like that for users in such a situation.]


Edit: Note to self - Need to come back and fill in details of work on 'overlapping samples' tests, especially overlapping samples t-tests


It occurs to me that a randomization test should work okay -

  • where the data are paired you randomly permute the group labels within pairs

  • where data are unpaired but assumed to have common distribution (under the null), you permute the group assignments

  • you can now base weights to the two shift estimates off the relative variance estimates ($w_1 = 1/(1+\frac{v_1}{v_2})$), compute each randomized sample's weighted estimate of shift and see where the sample fits into the randomization distribution.

share|improve this answer

Here are some thoughts. I basically just arrive to Greg Snow conclusion that This problem has distinct similarities to the Behrens-Fisher problem. To avoid handwaving I first introduce some notations and formalize the hypotheses.

  • we have $n$ paired observations $x_i^{pA}$ and $x_i^{pB}$ ($i = 1, \dots, n$);
  • we have $n_A$ and $n_B$ unpaired observations $x_i^A$ ($i = 1, \dots, n_A$) and $x_i^B$ ($i = 1, \dots, n_B$);
  • each observation is the sum of a patient effect and a treatment effect. The corresponding random variates are

    • $X_i^{pA} = P_i + T_i^A$, $X_i^{pB} = P_i + T_i^B$,
    • $X_i^A = Q_i + U_i^A$, $X_i^A = R_i + V_i^B$

    with $P_i, Q_i, R_i \sim \mathcal N(0,\sigma_P^2)$, and $T_i^\tau, U_i^\tau, V_i^\tau \sim \mathcal N(\mu_\tau, \sigma^2)$ ($\tau = A, B$).

    • under the null hypothesis, $\mu_A = \mu_B$.

We form as usual a new variable $X_i = X_i^{pA} - X_i^{pB}$. We have $X_i \sim \mathcal N(\mu_A - \mu_B, 2\sigma^2)$.

Now we have three groups of observations, the $X_i$ (size $n$), the $X_i^A$ (size $n_A$) and the $X_i^B$ (size $n_B$). The means are

  • $X_\bullet\sim \mathcal N(\mu_A - \mu_B, {2\over n} \sigma^2)$
  • $X^A_\bullet\sim \mathcal N(\mu_A , {1\over n_A} (\sigma_P^2 + \sigma^2))$
  • $X^B_\bullet\sim \mathcal N(\mu_B , {1\over n_B} (\sigma_P^2 + \sigma^2))$

The next natural step is to consider

  • $Y = X_\bullet + X^A_\bullet - X^B_\bullet \sim \mathcal N\left( 2(\mu_A-\mu_B), {2\over n} \sigma^2 + \left({1\over n_A}+ {1\over n_B}\right) (\sigma_P^2 + \sigma^2)\right)$

Now basically we are stuck. The three sums of squares give estimations of $\sigma^2$ with $n-1$ df, $\sigma_P^2 + \sigma^2$ with $n_A-1$ df and $n_B-1$ df respectively. The last two can be combined to give an estimation of $\left({1\over n_A}+ {1\over n_B}\right) (\sigma_P^2 + \sigma^2)$ with $n_A+n_B-2$ df. The variance of $Y$ is the sum of two terms, each of which can be estimated, but the recombination is not doable, just as in Behrens Fisher problem.

At this point I think one may plug-in any solution proposed to Behrens Fisher problem to get a solution to your problem.

share|improve this answer
I fixed a few typos in formulas. Please check! – kjetil b halvorsen Jun 12 '15 at 21:27
Thank you @kjetilbhalvorsen ! – Elvis Jun 13 '15 at 2:37

maybe mixed modelling with patient as random effect could be a way. With mixed modelling the correlation structure in the paired case and the partial missings in the unpaired case could be accounted for.

share|improve this answer
Since none of the datasets when analyzed individually would lead one to use random effects, I'm not seeing why random effects are helpful here. But it may be possible to use generalized least squares to allow each subject to have her own correlation structure. Unpaired observations would have correlation zero. Worth thinking about. Thanks. – Frank Harrell Apr 6 '12 at 3:04
yes, you're right, the datasets would not require mixed modelling if used separately. But if you append them into one single dataset, you could use the approach to incorporate the correlation in the paired data and simultaneously use the unpaired data by specifying a zero correlation. – psj Apr 7 '12 at 13:17
Yes; my point was that a mixed model may be an overkill as you can easily specify the subject-varying correlation structure using generalized least squares (using e.g. R's gls function in the nlme4 package. – Frank Harrell Apr 7 '12 at 15:51

My first thought was a mixed effects model, but that has already been discussed so I won't say any more on that.

My other thought is that if it were theoretically possible that you could have measured paired data on all subjects but due to cost, errors, or another reason you don't have all the pairs, then you could treat the unmeasured effect for the unpaired subjects as missing data and use tools like the EM algorithm or Multiple Imputation (missing at random seems reasonable unless the reason a subject was only measured under 1 treatment was related to what their outcome would be under the other treatment).

It may be even simpler to just fit a bivariate normal to the data using maximum likelihood (with the likelihood factored based on the available data per subject), then do a likelihood ratio test comparing the distribution with the means equal vs. different means.

It has been a long time since my theory classes, so I don't know how these compare on optimality.

share|improve this answer
Thanks Greg. I'm leaning towards the customized maximum likelihood approach. – Frank Harrell Jun 5 '12 at 18:12

One of the methods proposed in Hani M. Samawi & Robert Vogel (Journal of Applied Statistics , 2013) consists of a weighted combination of T-scores from independent and dependent samples in such a way that the new T score is equal to

$T_o = \sqrt\gamma ( \frac {\mu_Y - \mu_X} {S_x^2/n_X + S_y^2/n_Y}) + \sqrt {(1-\gamma)} \frac {\mu_D} {S_D^2/n_D}$

where $D$ represents the samples of paired differences taken from the correlated data. Basically the new T score is a weighted combination of the unpaired T-score with the new correction term. $\gamma$ represents the proportion of independent samples. When $\gamma$ is equal to 1 the test is equivalent to two sample t-test, whereas if equal to zero, it is a paired t-test.

share|improve this answer
Regarding this conservative weighted test for partially correlated data, the authors seem to distinguish several cases (large sample approximation, unequal ratio of matched/independent samples, small samples) in order to get the distribution of $T_0$ under the null. It would be helpful if you could summarize them in your answer, since the same article is referenced in another (partial) reply. – chl May 21 '14 at 11:50

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.