Comparing two means: Same group, different variance (Welch's t-test?)

Intro:

I have recently made a simulation program which simulates patients with Type 1 diabetes. In that context I am creating artifical patients. Let's denote three of these as $\text{p}_1$, $\text{p}_2$, and $\text{p}_3$. Assume that $\text{p}_1$, $\text{p}_2$ and $\text{p}_3$ are treated with treatment $A$. Treatment $A$ treats $\text{p}_1$, $\text{p}_2$ and $\text{p}_3$'s diabetes to a certain degree. We evaluate the treatment with some measures conducted in a virtual trial; let's denote one of these measures $\text{M}1_A$. $\text{M}1_A$ has an average value and a standard deviation for Treatment A for the three patients.

Now let's introduce treatment B for the same patient group, $\text{p}_1$, $\text{p}_2$ and $\text{p}_3$. Again we put the treatment through a trial test and receive a list of measures, this time denoted $\text{M}1_B$. I am interested in testing whether the difference in $\text{M}1_A$ and $\text{M}1_B$ is significant.

I know: $N=3, \text{std(M}1_A), \text{std(M}1_B), \text{mean(M}1_A), \text{mean(M}1_B)$

Question:

I want to compare the two measures tested on the same group of 'people' and use a statistical tool to figure out if the difference in the two measures are significant.

I am considering using Welch's t-test – would this be correct?

• WHen you say you know the means and standard deviations, do you not have the three original values for each treatment? – Glen_b Mar 18 '14 at 8:46
• I conduct the virtual trial on each patient for both treatments. This means that I can calculate the mean and standard deviation for the group on treatment A and the mean and standard deviation for the group on treatment B. @Glen_b I am not sure what you mean by original but, I would assume that yes i have the original values. – SteewDK Mar 18 '14 at 8:51
• If you're simulating, what makes the the results "on the same patient" rather than "on different patients" (i.e. what makes the same-patient measures 'more alike')? – Glen_b Mar 18 '14 at 9:24
• I am simulating so this means that treatment A will not effect treatment B. I assume that the treatments are done on the EXACT same patients - nothing differs between the two groups execpt the treatment itself. Thanks @Glen_b. – SteewDK Mar 18 '14 at 9:50
• If there's no effective difference in the simulation if you'd used patients p4,p5,p6 for the second treatment in your simulation, then it's not actually paired -- but if you do a real experiment, it matters a lot. – Glen_b Mar 18 '14 at 10:04

If they are the same three patients, the measurements are paired, in which case a Welch t-test is not ideal. A paired test of some kind, perhaps a paired t-test, a permutation test (that deals with the pairing), or a Wilcoxon signed rank test might be suitable, perhaps.

(You might also need to worry about order effects in your design.)

• Thanks for your reply. What would you use yourself? I am not that much into statistics so what exactly do you mean when you state that I should worry about order effects? How should I handle this? – SteewDK Mar 18 '14 at 8:14
• I don't know the circumstances; in some circumstances I might use any of those, depending on what exactly is the question of interest, what I know or feel safe assuming, or how badly the assumptions of the various procedures are likely to be wrong and how sensitive they are to those assumptions. In other circumstances I might do other things than those three. – Glen_b Mar 18 '14 at 8:41
• If you do an actual test of two treatments on a single patient, impact on the effect of the second treatment caused by being exposed to the first treatment before it would either be an order effect or a carry-over effect (which also matters). (ctd) – Glen_b Mar 18 '14 at 9:28
• (ctd) ... (Note that a paired design is a repeated measures design with only two measures.) -- the way to deal with order effects in practice is to balance the order of treatment (randomize patients half to one treatment first, half to the other treatment first). – Glen_b Mar 18 '14 at 9:33