3
$\begingroup$

I am running an experiment where I collect two data sets and I wish to measure the difference between the two. The two data sets are independent, with unknown probability distribution, and may not always have the same length.

Calculating the mean difference is easy as pie, but i also want a measure of the standard deviation and I'm not sure how to go about doing that. At first I just got the std deviation of the second data set minus the average of the first set, but in retrospect, I'm not sure that is entirely correct.

Any advice out there?

As an example of the data:

3.98   4.39   4.09   4.31   3.81   3.67   3.94   3.90   4.39   3.60   3.99   3.53   3.82

vs

3.95   4.51   4.49   4.43   4.55   4.41   4.68   4.22   4.45   4.59   4.42

Edit: I want to etch this in stone a bit more to explain why ttest would not be the answer I'm looking for: All computations are done in Matlab. note: var(one) = 3.61e-6, var(two) = 5.01e-06.
Using gui11aume's answer:

 std = 2.93e-3

when doing

 [h,p,ci,stats] =ttest2(one, two)
     stats.sd = 2.08e-3

realizing the variances are likely unequal, one should rewrite this to:

[h,p,ci,stats] =ttest2(one,two,0.05,'both','unequal')
    stats.sd =  [1.9e-3 2.23e-3]

once again, thank you for your time

$\endgroup$
  • 2
    $\begingroup$ I don't see the problem here. You can't take the difference of two data sets, only the difference between a function (like the mean) of one and the same function of the other. If there's some reason to disapprove of the mean (which I don't understand yet), there are other options (like the median). The standard error of the mean difference is the denominator of the t-test, which you can find anywhere. Can you state your final goal & why you don't want to work w/ the difference in means / t-test? $\endgroup$ – gung - Reinstate Monica May 25 '12 at 21:25
  • $\begingroup$ I think you need to tell us when you are talking about when you say you want to measure the standard deviation. I am as confused as gung. What you seem to be calculating does not resemble any statndard deviation that I know. You have no reason to pair the data and certainly can't when the sample sizes are different. Why not just calculate the standard deviation of the the difference between means. $\endgroup$ – Michael R. Chernick May 25 '12 at 21:47
  • $\begingroup$ In general it would be s1^2 +s2^2 -2 Cov(m1, m2) where s1 is the sample standard deviation from the sample from population 1 and s2 is the sample standard deviation from the sample from population 2 and Cov(m1, m2) is the covariance between the sample estimates m1 and m2. Since the samples are independent the covariances are 0. So just add the separate sample variances for population 1 and population 2. $\endgroup$ – Michael R. Chernick May 25 '12 at 21:49
  • $\begingroup$ @gung The mean is fine but doesn't provide a complete perspective. I'm not trying to prove that there is a difference (a fact I'm certain of), but trying the measure the difference in outcomes between these two sets, and a SD helps quantify that value. $\endgroup$ – Rasman May 25 '12 at 21:52
  • $\begingroup$ You aren't obligated to convert your t-score into a p-value, you can just leave it as $t$. The t-score is the difference b/t the 2 means standardized relative to the SD of the sampling distribution of differences. Alternatively, you could standardize the mean difference relative to the pooled SD of the data distributions, under the assumption of homogeneity of variance, this is the square root of the weighted average of the variances (w/ group df's over total df as weights) & is called 'Cohen's d'. $\endgroup$ – gung - Reinstate Monica May 25 '12 at 22:05
10
$\begingroup$

If $X$ and $Y$ are independent, the variance of $X-Y$ is $Var(X) + Var(Y)$. So the variance of the difference of means is the sum of the variances of each mean. This variance is unknown, but you can estimate it easily by the sum of the estimated variances: $S_1^2/n_1 + S_2^2/n_2$.

$\endgroup$
  • $\begingroup$ so you're saying if just compute the variance of each set independently, I can add the two and it will show the spread of my difference? I guess it kindda makes sense as I'm doing E(X) - E(Y) $\endgroup$ – Rasman May 25 '12 at 21:57
  • $\begingroup$ you're right, I'm doing a cross-correlation of two indepedant pdfs. I thought it was a simple problem, just couldn't figure out what I was doing wrong. $\endgroup$ – Rasman May 25 '12 at 22:07
  • $\begingroup$ gui11aume's answer is correct I meant to divide the standard deviations in my remarks by the sample sizes as he did so that it is an estimate of the variance of the mean difference. We get this answer because Cov(X,Y)=0 as would appear in the general formula before assuming independence. Then replacing Cov(X,Y) with 0 gives gui11aume's formula. $\endgroup$ – Michael R. Chernick May 25 '12 at 23:04
  • $\begingroup$ @Rasman said "I can add the two and it will show the spread of my difference?" -- note that variance is not the same as spread! $\endgroup$ – Glen_b -Reinstate Monica May 26 '12 at 4:57
  • 1
    $\begingroup$ @Rasman in this case the relevant measure of spread would be the square root of the sum of the variances $\endgroup$ – Glen_b -Reinstate Monica May 30 '12 at 23:16
0
$\begingroup$

There are better ways to estimate the standard deviation of the difference of two independent emperical datasets, when the probability distributions are unknown. You can use convolution, but a more simple nonparametric method is to use all possible differences between the two variables.

For a detailed explanation see: Poe, G. L., Giraud, K. L., & Loomis, J. B. (2005). Computational methods for measuring the difference of empirical distributions. American Journal of Agricultural Economics, 87(2), 353-365.

In R (sorry, I am not fluent in Matlab, but the conversion should not be too difficult):

     a=c(3.98, 4.39, 4.09, 4.31, 3.81, 3.67, 3.94, 3.90, 4.39, 3.60, 3.99, 3.53, 3.82)
     b=c(3.95, 4.51, 4.49, 4.43, 4.55, 4.41, 4.68, 4.22, 4.45, 4.59, 4.42)

     vardiff.est = var(a)+var(b) # valid for independent normal distributions
     (sd = sqrt(vardiff.est))

     # non-parametric estimation
     o=outer(a,b,'-')
     sd(o)  # vardiff.est seems to be close enough
     mean(o)
     hist(o, freq=F)
     curve(dnorm(x, mean(a)-mean(b), sd)), add=T) # a normal approximation seems to be fitting
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.