1
$\begingroup$

We have a client who manufactures materials with different surface 'roughness' characteristics, ranging from near-glass smooth to coarse sandpaper-rough.

They have ten different grades of roughness, from 1 (smooth) to 10 (coarse), and their engineers are trained to be able to identify each specific grade by touch.

Engineers undergo periodic training sessions, in which they have to identify the roughness grade of 10 random samples. Each engineer's responses are entered into our software, which produces a Pearson Correlation Coefficient as a measure of each engineer's performance in each test session. We supply the client with these results - i.e. a table resembling the following:

Training results table

Questions:

  1. Is it valid for the client to calculate the mean of the values for (e.g.) Session 1, in order to say that the average score for all engineers in Session 1 was (0.98 + 0.87 + 1.00) / 3 = 0.95?

  2. Likewise, is it valid for the client to calculate the mean for each engineer, in order to say that (e.g.) the average score for Helen Jones across the two sessions was (1.00 + 0.88) / 2 = 0.94?

Our client is not planning to use these numbers for any serious statistical analysis, but rather just to get a feel for whether the training sessions are helping the engineers to get better at judging the roughness of the materials.

Edit 1

Only integers are allowed for the engineer's responses. The correct answers correspond to defined roughness standards measured on a scale of 1 to 10.

I've looked again at the client's original spec. They are not looking to determine if the engineer can identify a specific roughness number for each sample (my mistake for stating this in my question). Rather, if the engineer can consistently distinguish rougher samples from smoother samples, then this should lead to a higher score.

So, if in a particular session the 'correct' answers are 1, 2, 3, 4, and John answers 1, 2, 3, 4 and Helen answers 2, 4, 6, 8, the fact that they both result in a correlation coefficient of 1 is fine for the client - they have both demonstrated that they score higher values for rougher samples.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

The correlations appear to be measuring the wrong quantity in the first place, so the question may be asking about how to summarize something that doesn't and can't perform as desired.

As I understand it, the correlations are measured for each engineer's reports as one variable and some independently obtained value for the true or correct roughness for the other variable. I don't gather from the question whether non-integers are allowed for either variable.

But a strong correlation could arise if an engineer were biased but otherwise reliable, e.g. if John's grades are given exactly by true $-\ 0.5$ or Helen's by true $+\ 0.5$ then their correlations would be $+1$. Conversely, any engineer for who there was deviation from engineer's $=$ true could not score a perfect correlation, even if they were on average unbiased. Otherwise put, correlation measures linearity of relationship, not agreement of values. The fallacy is seen most starkly by realising that $y$ and $by$ have perfect correlation for any positive value of $b$.

This may not bite very hard for your data, but if you desire a measure of agreement, then correlation can at best only be such a measure by accident. Concordance correlation is such a measure, although over-reliance on any single-valued reduction of any data is likely to miss important structure.

The forum contains various other threads mentioning concordance correlation, e.g. Does the concordance correlation coefficient make linearity or monotone assumptions?

In general, averaging correlations makes most sense when they encapsulate all the information of interest. Here that seems unlikely at best.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Many thanks for your answer. I've edited my question accordingly. Thanks also for pointing out the Concordance Correlation - this looks very interesting, and might be a good refinement in this case in order to get a higher quality report from the data. $\endgroup$
    – SLP
    Commented Jun 23, 2015 at 13:57
  • $\begingroup$ Thanks for the detail. Note that the constraints imply tilt, e.g. if the correct answer is 1, wrong answers can only be 2, 3, ... but not 0; with a remark of similar flavour but opposite sign for 10. This has implications for the structure of error. Some people would point you to kappa here as a measure of agreement. $\endgroup$
    – Nick Cox
    Commented Jun 23, 2015 at 14:02
  • $\begingroup$ The client seems to believe that correlation is the answer, regardless of the question. Correlation is just a measure summarizing one aspect of a relationship; there is no way that it can capture all the information in the data. Sooner or later that attitude will burn or bite them. Impute here what you should say when the client is being wrong-headed and you (presumably the consultant) are advising. $\endgroup$
    – Nick Cox
    Commented Jun 23, 2015 at 14:07
  • $\begingroup$ You're right Nick, the client is looking for a simple way of representing whether each engineers' scores are improving across training sessions - ideally represented as a single number per session. The correlation coefficient was their original choice, but I'm going to suggest looking at other more suitable alternatives such as the concordance correlation. $\endgroup$
    – SLP
    Commented Jun 23, 2015 at 15:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.