We have a client who manufactures materials with different surface 'roughness' characteristics, ranging from near-glass smooth to coarse sandpaper-rough.
They have ten different grades of roughness, from 1 (smooth) to 10 (coarse), and their engineers are trained to be able to identify each specific grade by touch.
Engineers undergo periodic training sessions, in which they have to identify the roughness grade of 10 random samples. Each engineer's responses are entered into our software, which produces a Pearson Correlation Coefficient as a measure of each engineer's performance in each test session. We supply the client with these results - i.e. a table resembling the following:
Is it valid for the client to calculate the mean of the values for (e.g.) Session 1, in order to say that the average score for all engineers in Session 1 was (0.98 + 0.87 + 1.00) / 3 = 0.95?
Likewise, is it valid for the client to calculate the mean for each engineer, in order to say that (e.g.) the average score for Helen Jones across the two sessions was (1.00 + 0.88) / 2 = 0.94?
Our client is not planning to use these numbers for any serious statistical analysis, but rather just to get a feel for whether the training sessions are helping the engineers to get better at judging the roughness of the materials.
Only integers are allowed for the engineer's responses. The correct answers correspond to defined roughness standards measured on a scale of 1 to 10.
I've looked again at the client's original spec. They are not looking to determine if the engineer can identify a specific roughness number for each sample (my mistake for stating this in my question). Rather, if the engineer can consistently distinguish rougher samples from smoother samples, then this should lead to a higher score.
So, if in a particular session the 'correct' answers are 1, 2, 3, 4, and John answers 1, 2, 3, 4 and Helen answers 2, 4, 6, 8, the fact that they both result in a correlation coefficient of 1 is fine for the client - they have both demonstrated that they score higher values for rougher samples.