0
$\begingroup$

Newbie to statistical implementations here; as I understand it, these are different. Please correct me where I am wrong.

One takes the pairwise difference of each point of data [ the mean of the differences ] and the other takes mean A and subtracts it from mean B [ the difference of the means ]. While the differences can be calculated to come out the same, the confidence intervals for each are different. I am confused as to which formula to use for which situation.

For example, I have a set of data with people from sample X, Y, or Z. There is one continuous variable, let's call it "communication skills", which goes from 0 - 100. I would like to test to see if the samples are statistically different from each other, with 95% confidence.

The two tests give two different confidence intervals. Which test would I use in this situation?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
3
$\begingroup$

Very simply, the MoD is used for paired samples (and paired t-tests), while the DoM is used for independent samples (and 2-sample t-tests).
While a previous answer is correct to say that the observed Mod will always be equal to the observed DoM, the CI's will indeed be different, with the CI of the MoD always narrower than the CI of the DoM. The intuitive reason being that for the MoD, we have 2 sources of uncertainty, the observed MoD (instead of the true population mean), and the observed sd (instead of the true population sd). But for the DoM, we have 4 such sources of uncertainty; the 2 observed means, and the 2 observed sd's (instead of the 4 corresponding true population parameters). Therefore, the DoM's CI will be wider.
Mathematically, the reason is that for the MoD, you are essentially performing a 1-sample t-test (on the paired differences). But the DoM is for a 2 sample t-test. And the increased CI width is due to how the variance is computed: the variance of the DOM will be larger.
It can be shown (e.g. here for a fuller derivation) that the variance of the DoM is $S^2=\dfrac {S^2_x+S^2_y} {n-1}$ (this is simply the variance of a difference of 2 samples), while the variance of the MoD is $S^2=\dfrac {S^2_x+S^2_y-2S_{xy}} {n-1}$ where $S_{xy}$ is the sample covariance between X and Y. Hence the narrower CI...

Now, when can you/should you use one or the other? You can use the paired t-test (which is preferred, because it gives narrower CI's, thus has greater power) if and only if the 2 samples are indeed paired, i.e. are dependent. The same "units" were measured for both the X and Y samples (maybe before and after some "intervention"). This also obviously implies that the 2 samples have the same size.
But if your 2 samples are independent (e.g. the "units" between the 2 samples are distinct, and independent), you have to use a 2-sample t-test. And the 2-sample t-test can handle the case where the 2 samples have different sizes.
Note that if there were no natural pairing between $X$ and $Y$, then I could arbitrarily re-order the 2 samples, and while the MoD would remain the same, the variances of these differently re-ordered samples would be different (because the $S_{xy}$ term would be different); so I would be able to obtain a whole range of t statistics, and hence results to my t-test. That should sound completely wrong...
You may find this paper or this CV post as good references.

Improve this answer
$\endgroup$
3
  • $\begingroup$ This is immensely helpful (despite the verbosity, haha). A follow up question, let's say I want to test the conversion rate within my population. One subset uses 2 products, the other uses 4 products. The conversion rate for the 4 product subset is much higher. Would this be a paired-samples test, as they both measure conversion rate, or not, because they are different subsets of the population? -> essentially, how does this work with subsets of populations? $\endgroup$
    – user443087
    Commented Nov 5 at 14:20
  • $\begingroup$ am I to understand, from the subtext, that it's quite easy to get confused as to which to use, when? $\endgroup$
    – user443087
    Commented Nov 5 at 14:29
  • $\begingroup$ Your example of conversion rate accross 2 different sets of subjects is not suitable for a paired test. Look up the definition for statistical "unit" (e.g. here en.wikipedia.org/wiki/Statistical_unit). For paired tests to be used, the units need to be the exact same in both groups. If they are different (your conversion case), you have to use a 2-sample test. Example of paired situation is measuring the same subjects before and after a "treatment" (to test the effect of the "treatment") $\endgroup$
    – jginestet
    Commented Nov 5 at 17:57
0
$\begingroup$

Suppose you have a sample of two data series each of length $n$, $\{x_i, y_i\}$.

The "mean of differences" is

$${\rm MoD} = \frac 1 n \sum_{i=1}^n (x_i-y_i) = \frac 1 n \sum_{i=1}^n x_i-\frac 1 n \sum_{i=1}^n y_i = {\rm DoM}.$$

Namely, it is exactly the same as the "difference of means". And if they are mathematically exactly the same, then, when viewed as random variables, they are one and the same (not just "having the same distribution", but one and the same).

Where did you find that "the confidence intervals are different"? It appears that you want to apply a "two-sample t-test". Maybe there is more to the situation?

Improve this answer
$\endgroup$
3
  • $\begingroup$ The original question came from learning the formulas for the "Confidence interval for the mean of a population" and the "confidence interval for the difference of two means" right beside one another, by author of the book also raises the question as well (it's one of those "for dummies" books, so I may missing something here, haha). The author states that """...Also note that there is a difference between the "difference in the means" and the "mean of the differences". If you're looking at pairs of data (such as pre-test versus post-test)... use the confidence interval for the mean. ... $\endgroup$
    – user443087
    Commented Nov 5 at 13:10
  • $\begingroup$ """ ... [otherwise] if you're examining the difference in the means of two separate populations (such as males versus females), use the methods in this section to find a confidence interval for the difference of two means. """ and logically, there must be a difference between CIs "R", and CI "V", if CIs R are confidence intervals for each array X and Y, and CI V is just the confidence interval of X-Y? No? The is what I am trying to test: If the mean of two sub-samples of a population are statistically different. I am using confidence intervals with t-scores, but am unsure which of above. $\endgroup$
    – user443087
    Commented Nov 5 at 13:14
  • $\begingroup$ @plotmaster473 As I suspected, there are more aspects in the situation which were not described in the original post. For example, to state the simplest one: difference in sample sizes. And indeed, as another answer pointed out there may be a difference in the standard deviation that is used to form the Confidence interval. $\endgroup$
    – Alecos Papadopoulos
    Commented Nov 5 at 16:29
0
$\begingroup$

To put it very simply, you use the mean of differences, when there is a natural pairing between your 2 groups. eg you give people a new toothpaste to try out and you compare the difference before and after using the toothpaste (number of caries).

Clearly there's a lot of variation between people - genetics, toothbrushing standard etc. So by taking the difference before and after for each person, we remove that variation ("noise") from the comparison and so our CIs are narrower.

If there is no such pairing - eg you give one group of people the toothpaste and another group continue to use their own toothpaste, then you can only use the difference in the means, and you aim to ensure you take a large enough sample size that those individual variations are averaged out. Precisely because of those individual variations the CI will be wider.

For example, I have a set of data with people from sample X, Y, or Z. There is one continuous variable, let's call it "communication skills", which goes from 0 - 100. I would like to test to see if the samples are statistically different from each other, with 95% confidence.

I assume you are interested in difference between sample X and Y. So as a "rule of thumb" if they are different people/subjects in groups X and Y, then you use difference of means, whereas if they are the same people ( eg X is before Training, Y is after communication Training), then you would use mean of differences.

The main reason for using paired tests is you have the same subjects but do before/after interventions or in different contexts.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ This is very helpful! So it seems like it's it's the same group, then I would use MoD, and if they are of different groups, then I would use DoM. Could you confirm the right formula for each of these? I have the MoD as xbar +/- ( z* ( s / np.sqrt(n) ) ), and the DoM as xbar- ybar +/- z * np.sqrt( (s2/n) + (s2/n) ). Thank you! $\endgroup$
    – user443087
    Commented Nov 5 at 15:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.