2
$\begingroup$

I have two sets of measurements from two different proxies (for simplicity we can assume one to be the gold standard). I want to report a number representing their agreement, ideally with 1.0 representing perfect agreement. They are each continuous variables in the same units having log-normal distribution.

I have used cor() in R, but this doesn't take into account bias, e.g. cor(x,x/2-1000) = cor(x,x) = 1. RMSE doesn't convey an intuitive sense of agreement or disagreement.

Lin's concordance seems like a good choice, but I am unsure how to use it properly. There are several CCC packages for R with many different options.

Improve this question
$\endgroup$
6
  • $\begingroup$ @lmo Perhaps; is there a way to migrate the question, or should I just re-ask the question there? $\endgroup$
    – teadotjay
    Commented Jun 18, 2016 at 17:49
  • $\begingroup$ I usually use cosine similarity. $\endgroup$
    – Bryan Goggin
    Commented Jun 18, 2016 at 17:58
  • $\begingroup$ @T.J. We can vote to migrate you, but if you want to delete and re-ask that's acceptable. Right now you've got 3 votes for migration, I'll add a 4th. $\endgroup$
    – Hack-R
    Commented Jun 18, 2016 at 18:06
  • $\begingroup$ @BryanGoggin I think cosine similarity has the same problem as Pearson's wrt. bias. Multiplicative biases yield identical values, and even very large additive biases yield high (>0.9) values of cosine similarity. $\endgroup$
    – teadotjay
    Commented Jun 18, 2016 at 20:16
  • $\begingroup$ Can you explain more about how your intuitive sense operates (in a way that would make it clear why/how it excludes RMSE)? $\endgroup$
    – Glen_b
    Commented Jun 19, 2016 at 0:16

2 Answers 2

Reset to default
3
$\begingroup$

Lin's concordance correlation is an appropriate single measure of agreement. Naturally it will not, and cannot, tell you much about the fine structure of agreement and disagreement.

With data that are (roughly or exactly) lognormal, the same question arises as with correlation, whether comparison on logarithmic scale makes as much or more sense than comparison on the original scale.

I can't help with your difficulties in deciding how to do it in R, which are not stated specifically and arguably are off-topic here in any case. If you could post sample data, alternative solutions could be explored.

Relevant threads here include

Does concordance correlation require data to be normally distributed?

Does the concordance correlation coefficient make linearity or monotone assumptions?

Improve this answer
$\endgroup$
2
  • $\begingroup$ This helps, thanks! Just one question to clarify. The decision of whether to calculate CCC on log scale depends on whether proportional error matters more than absolute error, rather than the distribution of data and errors? (If I recall the error distribution is roughly normal, or maybe skewed a bit if one set is biased). $\endgroup$
    – teadotjay
    Commented Jun 18, 2016 at 20:04
  • $\begingroup$ That could be happening, but I doubt that everything pivots on that alone. $\endgroup$
    – Nick Cox
    Commented Jun 18, 2016 at 22:12
2
$\begingroup$

Use McNemar's test to evaluate marginal homogeneity.

Use the poly/tetra-choric correlation coefficient if its assumptions are sufficiently plausible.

Possibly test association between proxies with the log odds ratio.

For more agreement tests with continuous variables check this out: http://web1.sph.emory.edu/observeragreement/review_manuscript.pdf

enter image description here

To assess raw agreement you can utilize Cohen's kappa for categorical data:

  • from its p-value, establish that agreement exceeds that expected under the null hypothesis of random ratings;
  • interpret the magnitude of kappa as an intraclass correlation.

In R you can use the psych package (cohen.kappa function).

Alternatively, calculate the intraclass correlation directly instead of a kappa statistic.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for the response. How would I use Cohen's kappa for continuous variables? I thought it applied only to categorical terms. I don't think my data fit the assumptions for polychoric correlation, unfortunately, as if I recall the distributions are skewed right even on a log scale. $\endgroup$
    – teadotjay
    Commented Jun 18, 2016 at 19:55
  • $\begingroup$ @T.J. Yes, you're correct it's for categorical variables. Sorry I didn't catch that you were using continuous but you should be able to use some of the other techniques mentioned and this should help you quite a lot -- web1.sph.emory.edu/observeragreement/review_manuscript.pdf I also updated my answer a bit. $\endgroup$
    – Hack-R
    Commented Jun 18, 2016 at 21:05
  • $\begingroup$ Thanks! I think the table will be useful, though I'll take me a while to digest that manuscript. :) $\endgroup$
    – teadotjay
    Commented Jun 18, 2016 at 21:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.