I'm analysing classroom interaction data. I'm using a simple quantitative index counting the number of interactions of a certain kind. I want to know if it significantly changed between years 1 and 2, where I used method A to teach, and year 3, where I used method B.

I thus have $x_1$ and $x_2$ from the first two years to compare against a single value $x_3$ for the third year.

It is correct to use a $t$-test to check if $x_3$ is significantly different from $x_1$ and $x_2$ by checking if, following a $t$ distribution with 1 degree of freedom, the probability of getting a value at least as extreme as $x_3$ is lower than some $p$-value?

That is, whether this holds:

$$P_{t(df=1)}\left(X \geq \frac{x_3 - \bar{x}}{s}\right) \leq 0.05,$$

where $\bar{x} = \frac{1}{n}\sum_{i=1}^n x_n$ and $s=\sqrt{\frac{1}{n-1}\sum_{i=1}^n (x_i-\bar{x})^2}$, so in this case where $n=2$ we would have $\bar{x} = \frac{x_1 + x_2}{2}$ and $s=\sqrt{(x_1 - \bar{x})^2 + (x_2 - \bar{x})²}$.

Is this correct, or shall I rather use $\frac{x_3 - \bar{x}}{s/\sqrt{2}}$ instead of $\frac{x_3 - \bar{x}}{s}$; i.e., divide by the standard error of the mean instead of the sample standard deviation?

(I am aware that this makes the assumption that the population follows a normal distribution.)

  • $\begingroup$ Do you only have 3 observations, or does $x_i$ represent a mean number of interactions from year $i$? If these are just three values, you don't have enough data to look for significance. $\endgroup$ – dankernler May 4 '18 at 11:46
  • $\begingroup$ Each of those values represents the mean number of interactions per student; I thus have three means to compare. $\endgroup$ – Jean-Philippe Pellet May 4 '18 at 17:00
  • 1
    $\begingroup$ In that case, your sample size is the number of students, not the number of samples. You want ANOVA if you want to test between all three years, or you can do a t-test if you want to lump the first two years together as one sample and compare that sample to year 3. $\endgroup$ – dankernler May 4 '18 at 17:32
  • $\begingroup$ But why couldn't I look for significance with three values? The first two give a sample mean and standard deviation. And I can compute the probability of a value at least as extreme as $x_3$ given this, can't I? $\endgroup$ – Jean-Philippe Pellet May 4 '18 at 19:52

If you just want to look for improvement from years 1 and 2 to year 3, then you could treat years 1 and 2 as a single sample and do a t-test. That sounds like what you want to do.

If it's helpful, you can check out this page I made for my intro stats class for comparing means from two independent samples.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.