4
$\begingroup$

I'm analysing classroom interaction data. I'm using a simple quantitative index counting the number of interactions of a certain kind. I want to know if it significantly changed between years 1 and 2, where I used method A to teach, and year 3, where I used method B.

I thus have $x_1$ and $x_2$ from the first two years to compare against a single value $x_3$ for the third year.

It is correct to use a $t$-test to check if $x_3$ is significantly different from $x_1$ and $x_2$ by checking if, following a $t$ distribution with 1 degree of freedom, the probability of getting a value at least as extreme as $x_3$ is lower than some $p$-value?

That is, whether this holds:

$$P_{t(df=1)}\left(X \geq \frac{x_3 - \bar{x}}{s}\right) \leq 0.05,$$

where $\bar{x} = \frac{1}{n}\sum_{i=1}^n x_n$ and $s=\sqrt{\frac{1}{n-1}\sum_{i=1}^n (x_i-\bar{x})^2}$, so in this case where $n=2$ we would have $\bar{x} = \frac{x_1 + x_2}{2}$ and $s=\sqrt{(x_1 - \bar{x})^2 + (x_2 - \bar{x})²}$.

Is this correct, or shall I rather use $\frac{x_3 - \bar{x}}{s/\sqrt{2}}$ instead of $\frac{x_3 - \bar{x}}{s}$; i.e., divide by the standard error of the mean instead of the sample standard deviation?

(I am aware that this makes the assumption that the population follows a normal distribution.)

$\endgroup$
4
  • $\begingroup$ Do you only have 3 observations, or does $x_i$ represent a mean number of interactions from year $i$? If these are just three values, you don't have enough data to look for significance. $\endgroup$
    – dankernler
    Commented May 4, 2018 at 11:46
  • $\begingroup$ Each of those values represents the mean number of interactions per student; I thus have three means to compare. $\endgroup$ Commented May 4, 2018 at 17:00
  • 2
    $\begingroup$ In that case, your sample size is the number of students, not the number of samples. You want ANOVA if you want to test between all three years, or you can do a t-test if you want to lump the first two years together as one sample and compare that sample to year 3. $\endgroup$
    – dankernler
    Commented May 4, 2018 at 17:32
  • $\begingroup$ But why couldn't I look for significance with three values? The first two give a sample mean and standard deviation. And I can compute the probability of a value at least as extreme as $x_3$ given this, can't I? $\endgroup$ Commented May 4, 2018 at 19:52

2 Answers 2

1
$\begingroup$

If you just want to look for improvement from years 1 and 2 to year 3, then you could treat years 1 and 2 as a single sample and do a t-test. That sounds like what you want to do.

If it's helpful, you can check out this page I made for my intro stats class for comparing means from two independent samples.

$\endgroup$
1
$\begingroup$

I would use a linear mixed-effects model with random effects to account for non-independence caused by multiple measurements on individual students. I would probably treat year as a categorical fixed effect in this case, though there may be an argument for replacing year with teaching_method.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.