You could use an analysis of variance (ANOVA) to do null hypothesis significance testing (NHST) or to calculate an intraclass correlation coefficient (ICC). Both approaches are based on the same ANOVA, but they differ in several important ways. I will argue that these differences make the ICC a preferable option, but you may disagree. The important thing is to understand the consequences of whichever choice you make. In short, the NHST approach will only tell you if the raters' measurements are significantly different; it won't tell you how different they are or how the variance associated with raters compares to the variance associated with other sources of variance.
I will refer to "objects of measurement" and "raters." Objects are the things being measured (e.g., emails, films, or X-Ray images) and raters are the things providing these measurements (e.g., spam filtering algorithms, film critics, or radiologists). The measurements themselves may be nominal categories (e.g., spam or non-spam), ordinal categories (e.g., 1, 2, 3, 4, or 5 stars), or numbers on an interval/ratio scale (e.g., tumor diameter in mm).
If you use a one-way model, the ANOVA will yield two mean squares: one for object of measurement and one for residual sources of variance. Because there is no mean square for rater, any variance that is related to differences between raters will be collapsed into the residual term along with variance due to measurement error and other unaccounted for variance. Thus, an NHST based on a one-way model is somewhat confounded in the application you seem to be describing.
If you use a two-way model, the ANOVA will yield three mean squares: one for object of measurement, one for rater, and one for residual sources of variance. These mean squares can be used in NHST or to calculate an ICC. This type of model is typically more useful for analyzing inter-rater reliability.
To concretize things, let's look at some example data of six objects measured by four raters on a 1 to 10 interval scale, presented in the following object-by-rater matrix:
$$
\begin{array}
. & R1 & R2 & R3 & R4 \\
O1 & 9 & 2 & 5 & 8 \\
O2 & 6 & 1 & 3 & 2 \\
O3 & 8 & 4 & 6 & 8 \\
O4 & 7 & 1 & 2 & 6 \\
O5 & 10 & 5 & 6 & 9 \\
O6 & 6 & 2 & 4 & 7 \\
\end{array}
$$
Using a two-way model, you get the following mean squares:
$MS_{Rater}=MSR=32.49$
$MS_{Object}=MSO=11.24$
$MS_{Error}=MSE=1.02$
To use these for NHST, you would calculate an $F$ statistic for the rater source of variance (as below), which we can compare to the critical $F_{(3,15)}=2.49$. Our observed $F$ exceeds the critical value, so we can say that the raters differ to a "statistically significant" degree. But that's about all...
$$F_{Rater}=\frac{MSR}{MSE}=\frac{32.49}{1.02}=31.87$$
To calculate an ICC, you would use the following formulas to calculate the degree of consistency (denoted $C$) or absolute agreement (denoted $A$) between each rater and each other rater (denoted $1$) or between the mean of all raters and each rater (denoted $k$). Absolute agreement requires all raters to have the same mean (e.g., assign the same exact number to an object), whereas consistency allows each rater to have their own mean as long as the raters covary together (e.g., see an object as relatively high or low).
$ICC(C,1)=\frac{MSO-MSE}{MSO+(k-1)MSE}=\frac{11.24-1.02}{11.24+(4-1)1.02}=0.72$
$ICC(A,1)=\frac{MSO-MSE}{MSO+(k-1)MSE+(k/n)(MSR-MSE)}=\frac{11.24-1.02}{11.24+(4-1)1.02+(4/6)(32.49-1.02)}=0.29$
$ICC(C,k)=\frac{MSO-MSE}{MSO}=\frac{11.24-1.02}{11.24}=0.91$
$ICC(A,k)=\frac{MSO-MSE}{MSO+(MSR-MSE)/n}=\frac{11.24-1.02}{11.24+(32.49-1.02)/6}=0.62$
So basically, the ICC gives you a quantification of reliability instead of a binary "yes" or "no" and, by virtue of its various formulations, allows you to control various aspects of what counts as reliability and what doesn't. By computing all four formulations, you can also get a sense of the raters' measurements. In this example, consistency is quite reasonable but agreement is a lot lower. So it would appear that the raters have differing means but seem to covary together pretty well. Visual inspection of the example data matrix can also give this sense, although checking a larger matrix would be more difficult. Additionally, the average score ICCs are a lot higher than the single score ICCs (which is always somewhat true) so a researcher could be a lot more comfortable using the average of all 4 raters than the measurements from any single rater.
References
McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30–46.
