I have 30 participants. They have a pre score and a post score. I am testing whether this changes. There are five observations per participant.

When the data are analyzed using a t-test there is a significant difference. When analyzed using linear mixed effects (i.e., with random subject and item intercepts) there is no significant difference (using the lmerTest package in R to generate p values via Satterthwaite's degrees of freedom method).

What conclusions can I draw from the difference between results?

| cite | improve this question | | | | |
  • 1
    $\begingroup$ Could you be more specific about how you are employing a t-test with such data? $\endgroup$ – whuber Nov 20 '19 at 23:08
  • $\begingroup$ @whuber Sure! The t-test would be on each participant's mean pre-training score compared to their mean post-training score. $\endgroup$ – Dave Nov 20 '19 at 23:18
  • $\begingroup$ Dave, can you please clarify how "five observations per participant" translates to "a pre score and a post score?" In typical use Pre/Post pretty specifically means two observations. $\endgroup$ – Alexis Nov 20 '19 at 23:57
  • $\begingroup$ @Alexis Sure! It was actually six observations per participant. Three of these were pre-intervention (i.e., measured our DV using three different stimuli pre intervention) and then three were post-intervention. $\endgroup$ – Dave Nov 20 '19 at 23:59
  • 1
    $\begingroup$ That doesn't sound like "a t-test:" it sounds like 30 t-tests. The difference isn't trivial, because it's crucial to compensate for making all those tests at once. Moreover, 30 tests do not assess whether there is a significant difference overall: it's making a determination about each participant separately. $\endgroup$ – whuber Nov 21 '19 at 15:12

Measurements on the same subjects are correlated. If you ignore these correlations, and you perform a simple t-test as you did, then you expect that

  • p-values are wrongly too small for between-subjects effects, and
  • p-values are wrongly too large for within-subjects effects

The reason is that the variance of correlated data does not equal to the variance of independent data, assumed by the standard (unpaired) t-test.

If interested, you may find more information in Section 1.2 of my course notes.

| cite | improve this answer | | | | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.