1
$\begingroup$

I have a sample of 15 subjects that did a test multiple times. Now I would like to see if the subjects differ in their mean performance. The usual approach would be a test for differences in means (e.g. t-test), but if I compare the subject with each other, I get a large type 1 error rate. Because of that, I thought about calculating a confidence interval for each subject and to see if they overlap.

What happens with the Type 1 Error here? Is there an alternative method to analyse the data?

Thanks for your patience and answers!

Improve this question
$\endgroup$
5
  • $\begingroup$ Looking at confidence intervals can be misleading. Sometimes two whiskers can overlap and difference still significant. Second, confidence interval and t-test are different products of the same process; if multiple testing is involved, threshold p is lowered, then the t-distribution constant that goes into the CI calculation should also be higher, making the CI wider. Aka the CI will not remain unchanged. $\endgroup$
    – Penguin_Knight
    Commented Feb 25, 2015 at 12:16
  • $\begingroup$ you mean that I should first correct the comparison-wise type 1 error and then use the corresponding t-value for the construction of the confidence interval? $\endgroup$
    – 00schneider
    Commented Feb 25, 2015 at 12:52
  • $\begingroup$ Yes and no, I meant to indicate that i) the confidence interval method also suffers from inflated type I error as well, and ii) because of that, you may as well go for t-test, but with additional adjustment to the p-value. Look for "Post-hoc mean comparison" and you may find some useful way to address the inflated type I error issue. $\endgroup$
    – Penguin_Knight
    Commented Feb 25, 2015 at 14:54
  • $\begingroup$ How many observations total? You have 15 subjects, how many total test observations? $\endgroup$
    – robin.datadrivers
    Commented Feb 25, 2015 at 15:24
  • $\begingroup$ each one did the test 5-6 times, not many, I know. Bad design? $\endgroup$
    – 00schneider
    Commented Feb 25, 2015 at 15:29

1 Answer 1

Reset to default
2
$\begingroup$

I would approach this as a repeated measures design, which can be done in a multilevel model framework which is robust to imbalance in the number of observations per subject.

Think about it this way:

$score_{it}=\beta_{i}+\mu_i+\epsilon_{it}$

or rewritten in a two-level framework:

$score_{it}=\beta_{i}+\epsilon_{it}$

$\beta_i=\gamma_i+\mu_i$

Where $\beta_i$ is the mean score for each subject across all their tests, $\mu_i$ is the random effect for the subject (i.e. the variability in score due to the subject), and $\epsilon_it$ is the random error for each individual test-subject unit (i.e. the within subject variability). This is flexible to put in additional level 1 or level 2 variables, like if you think subjects improve over time, you can add that in as well and it becomes a growth-curve model.

This doesn't suffer from multiple testing, and you can see how much variability is due to the subjects themselves, by the intra-class correlation coefficient (ICC): $\sigma_{\mu}/(\sigma_{\mu}+\sigma_{\epsilon})$

Improve this answer
$\endgroup$
2
  • $\begingroup$ Would the first $\beta_i$ actually be $\gamma_i$? $\endgroup$
    – Penguin_Knight
    Commented Feb 26, 2015 at 2:10
  • $\begingroup$ Correct; if collapse the two-level interpretation into the one-level, the beta and gamma are equivalent. $\endgroup$
    – robin.datadrivers
    Commented Feb 26, 2015 at 4:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.