2
$\begingroup$

I measured 30 samples each with 3 devices. The devices are the same make and model, just different units. How do I statistically test if the devices measurements are the same or not? If I had two data sets to compare, a simple paired t-test works great. I've read that running three paired t-tests (1vs2, 2vs3, and 3vs1) separately to draw conclusions is incorrect...the familywise error is higher than the error of the individual comparisons. But when I set-up ANOVA it doesn't pair the data by sample, it just lumps everything into a single average, and the effect of each device is lost.

So what test do I use?

Improve this question
$\endgroup$
2
  • $\begingroup$ You seem to be conflating the terms paired and pairwise. t-tests are pairwise when you have multiple groups that you compare to each other. They are paired if you have in each group the same number of measurements with matched sample points (within subjects). In your case, it is paired and pairwise, but the concepts are different. $\endgroup$
    – David Ernst
    Commented Oct 25, 2016 at 22:28
  • $\begingroup$ Thank you for explicitly pointing that out. It does get confusing. $\endgroup$
    – Bryan
    Commented Oct 25, 2016 at 22:48

2 Answers 2

Reset to default
2
$\begingroup$

You seem to be conflating the terms paired and pairwise. t-tests are pairwise when you have multiple groups that you compare to each other. They are paired if you have in each group the same number of measurements with matched sample points (within subjects). In your case, it is paired and pairwise, but the concepts are different.

There are within subject Anovas that consider the data paired. I'm not an expert here, since I can't use them in my experiments. In general, you should know that Anova + post hoc t-tests are often bundled for convenience, not for binding statistical reasons. The Anova itself cannot tell you very much. Only if there are significant differences among some of the groups or not. If you want information about specific pairs of groups, look at the t-tests.

Don't compute the differences before handing the data to an ANOVA or you will be testing for equality of differences where you want to test for equality of measures. (Unless the SAS documentation explicitly says so, which I don't believe.)

For the post-hoc tests, just subtract the two columns before entering the difference as one column into the t-test. In addition, you're doing pairwise t-tests which means that you also need to control the family wise error rate with a Bonferroni correction or Holm's method. Those two things are independent from each other and the t-tests are also independent from the ANOVA. So if you want to to the Anova and t-tests, better not use the Tuckeys tests, you will lose power. Even if they come with the Anova (which they shouldn't with a within subjects Anova), you can simply throw them out and do your own.

Improve this answer
$\endgroup$
2
0
$\begingroup$

One way ANOVA with a follow up Tukey Test would be a common way to go. The Tukey Test takes the ANOVA object and controls for the familywise error rate.

If you're working in R, you'll use the TukeyHSD function for this.

Improve this answer
$\endgroup$
6
  • $\begingroup$ I may be missing something, but the ANOVA doesn't consider my data to be paired. It is only looking at the mean and errors of the measurements from each device, like an unpaired t-test. $\endgroup$
    – Bryan
    Commented Oct 25, 2016 at 21:54
  • $\begingroup$ There are within subject ANOVAs that consider the data paired. For the post-hoc tests, just substract the two columns before entering the difference as one column into the ttest. In addition, you're doing pairwise t-tests which means that you also need to control the family wise error rate with a Bonferroni correction or Holm's method. Those two things are independent from each other and the t-tests are also independent from the ANOVA. $\endgroup$
    – David Ernst
    Commented Oct 25, 2016 at 22:31
  • $\begingroup$ After reading your comment and another post (stats.stackexchange.com/questions/115428/anova-for-paired-data), I should not analyze the measurements themselves but the DIFFERENCE among the measurements. I would have three calculated columns: m1-m2, m2-m3, and m3-m1. Then I would perform an ANOVA on these data and subsequently a post-hoc test. Without delving too much into another question, is there a reason Bonferroni or Holm's are particularly suited to this problem, as opposed to TukeysHSD? $\endgroup$
    – Bryan
    Commented Oct 25, 2016 at 22:50
  • $\begingroup$ Tuckey's test is not for paired samples. You can still use it I think, but you would lose some power. I think that within subjects ANOVA's take care automatically of computing the differences. Not entirely sure since I cannot use them in my experimental setup for another reason. I therefore do my multiple t-tests on one column with precomputed differences each and then Holm's procedure. No Anova, but the t-tests work perfectly fine without it. $\endgroup$
    – David Ernst
    Commented Oct 25, 2016 at 23:05
  • $\begingroup$ My bad, didn't actually notice you were working with paired samples. Sorry about that! $\endgroup$
    – Conor Neilson
    Commented Oct 25, 2016 at 23:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.