# Strange two sample z test result in experiment

I performed an experiment on a virtual population wanting to see whether Vitamin C can positively affect performance on an IQ test. I gathered 80 people and grouped them into two groups of 40.

First, I had them all take an IQ test. A day later, I gave half of them a vitamin C drink and the other half a sugar water drink as a placebo and then had them take an IQ test again.

I then performed a two-sample z test on the differences in scores in the control and treatment groups (This is how I accounted for the probable possibility that the people will do better on the second test after having learned from the first. If vitamin C does improve their scores, then the mean increase will be significantly higher than that of the control group regardless.)

The mean increase in score for the treatment group was 2.775, while the mean increase in the placebo group was 11.55. The z-score turned out to be ~ -16, yielding of course a tiny p-value. I am very confused and have no idea what conclusion to make. It seems to me that there is very strong evidence that vitamin C hurts IQ test performance, but that wasn't even what I was testing. What conclusion can I make with these surprising results?

• Why did you use a z test? Do you have the population, as opposed to sample standard errors to work with? Shouldn't you be conducting a t test? – Alexis Mar 13 '15 at 5:55
• You're right. I completely forgot that z only works with population sd. I have changed my test to a t-test. But I am still unsure as to how I can come to a conclusion based on the results I have. I've never encountered a situation like this before. Any help would be appreciated. – Gerk Mar 13 '15 at 6:11
• Can you share your data? Or at least your $\bar{x}_{\text{Vit C}}$, $\bar{x}_{\text{Sugar}}$, the standard errors for both and the sample sizes for each group? Could make sure its the math vs. needing to interpret the effect of a sugar rush on IQ. :) – Alexis Mar 13 '15 at 6:18
• Vit C group: n = 40, mean increase in score = 2.775, SD = 6.466829 (so SE ~ 1.0225) Sugar water group: n = 40, mean increase in score = 11.55, SD = 5.44883 (so SE ~ 0.8615) I used R to double check the means and standard deviations, so I’m fairly certain the math is fine. My stats teacher told me that sugar water is fine to use as a placebo with this virtual population, so let’s assume that it is not a sugar rush causing it. – Gerk Mar 13 '15 at 6:31

Given your data, an unpaired t test statistic (not assuming equal variances):

$$t = \frac{(\bar{x}_{\text{Vit C}} - \bar{x}_{\text{Sugar}})}{\sqrt{\left(\frac{s^{2}_{\text{Vit C}}}{n_{\text{Vit C}}}\right)+\left(\frac{s^{2}_{\text{Sugar}}}{n_{\text{Sugar}}}\right)}} = \frac{(2.775 - 11.55)}{\sqrt{\left(\frac{6.466829^{2}}{40}\right)+\left(\frac{5.44883^{2}}{40}\right)}} = -6.56,$$

with degrees of freedom (using Satherwaite's formula):

$$\nu = \frac{\left(s^{2}_{\overline{\text{Vit C}}} + s^{2}_{\overline{\text{Sugar}}}\right)^{2}}{\left(\frac{s^{4}_{\overline{\text{Vit C}}}}{n_{\text{Vit C}}-1} + \frac{s^{4}_{\overline{\text{Sugar}}}}{n_{\text{Sugar}}-1}\right)} = \frac{\left(1.0225^{2} + 0.8615^{2}\right)^{2}}{\left(\frac{1.0225^{4}}{40-1} + \frac{0.8615^{4}}{40-1}\right)} = 76$$

The $p$-value $= P(|T_{\nu=76}| \ge |-6.5|)=2.93\times10^{-9}$, so if $H^{+}_{0}:\mu_{\text{Vit C}} - \mu_{\text{Sugar}}=0$; and $H^{+}_{\text{A}}:\mu_{\text{Vit C}} - \mu_{\text{Sugar}}\ne 0$ we would reject $H^{+}_{0}$ for even very tiny levels of $\alpha$.

Well, that's all well and good, but frequentist tests are biased to reject the null hypothesis, and we do not know if the rejection decision for $H^{+}_{0}$ was due to being overpowered for an effect size that we consider relevant. Fortunately, within a frequentist hypothesis testing framework, we can combine inference about equivalence using the two one-sided tests approach to draw conclusions that explicitly account for power and effect size.

Let's say that the minimum difference between $\bar{x}_{\text{Vit C}}$ and $\bar{x}_{\text{Sugar}}$ that we would accept as relevantly large is, say 2 (you might actually care about a different minimum relevant effect size), and let's call this number $\Delta$. We can then pose:

$H^{-}_{0}: |\mu_{\text{Vit C}} - \mu_{\text{Sugar}}|\ge \Delta$, which translates into two one-sided test null hypotheses:

$H^{-}_{01}: \mu_{\text{Vit C}} - \mu_{\text{Sugar}}\ge \Delta$, –or–

$H^{-}_{02}: \mu_{\text{Vit C}} - \mu_{\text{Sugar}}\le -\Delta$

We can reject $H^{-}_{0}$ by rejecting both the two one-sided t tests associated with $H^{-}_{01}$ and $H^{-}_{02}$:

$$t_{1} = \frac{\Delta - (\bar{x}_{\text{Vit C}} - \bar{x}_{\text{Sugar}})}{\sqrt{\left(\frac{s^{2}_{\text{Vit C}}}{n_{\text{Vit C}}}\right)+\left(\frac{s^{2}_{\text{Sugar}}}{n_{\text{Sugar}}}\right)}} = \frac{2 - (2.775 - 11.55)}{\sqrt{\left(\frac{6.466829^{2}}{40}\right)+\left(\frac{5.44883^{2}}{40}\right)}} = 8.05,$$

and

$$t_{2} = \frac{(\bar{x}_{\text{Vit C}} - \bar{x}_{\text{Sugar}}) + \Delta}{\sqrt{\left(\frac{s^{2}_{\text{Vit C}}}{n_{\text{Vit C}}}\right)+\left(\frac{s^{2}_{\text{Sugar}}}{n_{\text{Sugar}}}\right)}} = \frac{(2.775 - 11.55)+2}{\sqrt{\left(\frac{6.466829^{2}}{40}\right)+\left(\frac{5.44883^{2}}{40}\right)}} = -5.067072.$$

The $p$-values of both these test statistics are obtained in the right tail, so $p_{1} = P(T_{\nu=76} \ge 8.05) = 4.25\times 10 ^{-12}$ and $p_{2} = P(T_{\nu=76} \ge -5.067072) = 0.9999986$. Since we do not reject both $H^{-}_{01}$ and $H^{-}_{02}$, we do not reject $H^{-}_{0}$.

We can then draw conclusions based on both our test for difference (i.e. test of $H^{+}_{0}$) and our test for equivalence (i.e. test of $H^{-}_{0}$), and because we rejected the former but not the latter we conclude a relevant difference (given $\alpha$ and $\Delta$). Because the difference test statistic is negative, we can indeed conclude that the group treated with Vitamin C improves its performance on IQ tests significantly (and relevantly) worse than the group given Sugar.

Your comment "My stats teacher told me that sugar water is fine to use as a placebo with this virtual population, so let’s assume that it is not a sugar rush causing it." is telling: you have made a decision a priori to evaluating the evidence, so one wonders why you are bothering to attempt statistical inference, when you have already decided your answer?