I want to compare 3 different tests applied to the same group of students at the same occasion. Is an ANOVA the right procedure? If so which type? And what exactly doess ANOVA give that a correlation won't? Thanks
Normal test scores, Block ANOVA. First, suppose the test scores are nearly normal. If you were comparing only two tests, then a paired t test would be the appropriate procedure. The generalization of 'pair' for more than two is 'block'. So you can do a block ANOVA. In this simple case, a block ANOVA is equivalent to a two-factor ANOVA, where the main effect is 'Test type' with levels A, B, C. The other factor 'Student' is random, with levels $1, 2, \dots, n,$ for the $n$ students in the group. Each of the $3n$ cells in the design will have one test score.
If significant, 'ad hoc' comparisons. Obviously, students will be of different abilities and one expects the Student effect to be significant, indicating that variability. Of main interest is whether the Test effect is significant. If so, you can make ad hoc comparisons among pairs of tests to see what the pattern of the differences may be. These should be paired tests, involving differences between, say test A and B, for each student. (You might use Tukey HSD or Bonferroni methods to avoid 'false discovery' for the three comparisons involved.)
You should check to see if the residuals look normal. If the number of students is large, an ANOVA may be OK even if residuals are not far from normal; large numbers of outliers may be a warning that all is not well.
Non-normal data, Friedman test. If you question whether test scores are normal, or if you have a very small group of students, you might use a nonparametric Friedman test. This test will give you a P-value only for Tests (but not for Students). If tests differ, you could use Wilcoxon signed rank tests to compare tests pairs A-B, A-C, B-C for significance, again with some method to control the 'family' error rate for the three comparisons.