Skip to main content
added 136 characters in body
Source Link
kjetil b halvorsen
  • 82.8k
  • 32
  • 201
  • 663

What you are looking for is agreement, not correlation. This is different: If comparing two sets of ratings from two raters (examiners), denote one set by $x$ the other $y$. If these ratings or scores satisfy $$ y=x+1, $$ say, then the correlation will be 1 but there is no agreement; in fact there is a constant disagreement of $|y-x| = 1$. Any measurement which does not take that into account cannot be an index of agreement! One correlation coefficient which could be used is the concordance correlation coefficient: see https://en.wikipedia.org/wiki/Concordance_correlation_coefficient This can be very close to some measures called ICC (there are many). t-test comparisons could be useful, but remember in the model $$ y_i = \alpha + \beta x_i + \epsilon_i $$ you want to test both $\alpha=0$ and $\beta=1$. This somehow, while it might be useful, seems artificial since it treats $x$ and $y$ differently, while the problem of agreement is really symmetric.

Coefficients like kappa or Krippendorff's alpha are directly constructed for evaluating agreement, so are probably the most natural choice. Krippendorff's alpha is the more general of the two, and, indeed, it can be used with more than two raters. There is an R package irr which implements these and others. But these coefficients are numbers and do not indicate the nature of the disagreement, so plots (specifically the Tukey mean-difference plot is very useful, also called Bland-Altman plot How does one interpret a Bland-Altman plot?, or Agreement between methods with multiple observations per individual). Regression models as above can also be useful in understanding the nature of the disagreement.

A book-length treatment is Kilem L. Gwet: "Handbook of interrater reliability" which I just got from the library for a project, so in some time maybe I will come back with some more information.

What you are looking for is agreement, not correlation. This is different: If comparing two sets of ratings from two raters (examiners), denote one set by $x$ the other $y$. If these ratings or scores satisfy $$ y=x+1, $$ say, then the correlation will be 1 but there is no agreement; in fact there is a constant disagreement of $|y-x| = 1$. Any measurement which does not take that into account cannot be an index of agreement! One correlation coefficient which could be used is the concordance correlation coefficient: see https://en.wikipedia.org/wiki/Concordance_correlation_coefficient This can be very close to some measures called ICC (there are many). t-test comparisons could be useful, but remember in the model $$ y_i = \alpha + \beta x_i + \epsilon_i $$ you want to test both $\alpha=0$ and $\beta=1$. This somehow, while it might be useful, seems artificial since it treats $x$ and $y$ differently, while the problem of agreement is really symmetric.

Coefficients like kappa or Krippendorff's alpha are directly constructed for evaluating agreement, so are probably the most natural choice. Krippendorff's alpha is the more general of the two, and, indeed, it can be used with more than two raters. There is an R package irr which implements these and others. But these coefficients are numbers and do not indicate the nature of the disagreement, so plots (specifically the Tukey mean-difference plot is very useful, also called Bland-Altman plot How does one interpret a Bland-Altman plot?). Regression models as above can also be useful in understanding the nature of the disagreement.

A book-length treatment is Kilem L. Gwet: "Handbook of interrater reliability" which I just got from the library for a project, so in some time maybe I will come back with some more information.

What you are looking for is agreement, not correlation. This is different: If comparing two sets of ratings from two raters (examiners), denote one set by $x$ the other $y$. If these ratings or scores satisfy $$ y=x+1, $$ say, then the correlation will be 1 but there is no agreement; in fact there is a constant disagreement of $|y-x| = 1$. Any measurement which does not take that into account cannot be an index of agreement! One correlation coefficient which could be used is the concordance correlation coefficient: see https://en.wikipedia.org/wiki/Concordance_correlation_coefficient This can be very close to some measures called ICC (there are many). t-test comparisons could be useful, but remember in the model $$ y_i = \alpha + \beta x_i + \epsilon_i $$ you want to test both $\alpha=0$ and $\beta=1$. This somehow, while it might be useful, seems artificial since it treats $x$ and $y$ differently, while the problem of agreement is really symmetric.

Coefficients like kappa or Krippendorff's alpha are directly constructed for evaluating agreement, so are probably the most natural choice. Krippendorff's alpha is the more general of the two, and, indeed, it can be used with more than two raters. There is an R package irr which implements these and others. But these coefficients are numbers and do not indicate the nature of the disagreement, so plots (specifically the Tukey mean-difference plot is very useful, also called Bland-Altman plot How does one interpret a Bland-Altman plot?, or Agreement between methods with multiple observations per individual). Regression models as above can also be useful in understanding the nature of the disagreement.

A book-length treatment is Kilem L. Gwet: "Handbook of interrater reliability" which I just got from the library for a project, so in some time maybe I will come back with some more information.

deleted 5 characters in body
Source Link
Nick Cox
  • 59.5k
  • 8
  • 136
  • 212

What you are looking for is agreement, not correlation. This is different: If comparing two sets of ratings from two raters (examiners), denote one set by $x$ the other $y$. If these ratings or scores satisfy $$ y=x+1, $$ say, then the correlation will be 1 but there is no agreement, ifagreement; in fact there is a constant disagreement of $|y-x| = 1$. Any measurement which dodoes not take that into account cannot be an index of agreement! One correlation coefficient which could be used, is the concordance correlation coefficient,: see https://en.wikipedia.org/wiki/Concordance_correlation_coefficient This can be very close to some measures called ICC (there are many). t-test comparisons could be useful, but remember in the model $$ y_i = \alpha + \beta x_i + \epsilon_i $$ you want to test both $\alpha=0$ and $\beta=1$. This somehow, while it might be useful, seems artificial since it treats $x$ and $y$ differently, while the problem of agreement is really symmetric.

Coefficients like kappa or Krippendorffs'Krippendorff's alpha are directly constructed for evaluating agreement, so isare probably the most natural choice. Krippendorffs'Krippendorff's alpha is the more general of the two, and  , indeed, it can be used with more than two raters. There is an R package irr which implements these and others. But these coefficients is only a number, it doesare numbers and do not indicate the nature of the disagreement, so plots (specifically the Tukey mean-difference plot is very useful, also called Bland-Altman plot How does one interpret a Bland-Altman plot?). Regression models as above can also be useful in understanding the nature of the disagreement.

A book-length treatment is Kilem L. Gwet: "Handbook of interrater reliability" which I just got from the library for a project, so in some time maybe I will come back with some more information.

What you are looking for is agreement not correlation. This is different: If comparing two sets of ratings from two raters (examiners), denote one set by $x$ the other $y$. If these ratings or scores satisfy $$ y=x+1, $$ say, then the correlation will be 1 but there is no agreement, if fact there is a constant disagreement of $|y-x| = 1$. Any measurement which do not take that into account cannot be an index of agreement! One correlation coefficient which could be used, is the concordance correlation coefficient, see https://en.wikipedia.org/wiki/Concordance_correlation_coefficient This can be very close to some measures called ICC (there are many). t-test comparisons could be useful, but remember in the model $$ y_i = \alpha + \beta x_i + \epsilon_i $$ you want to test both $\alpha=0$ and $\beta=1$. This somehow, while it might be useful, seems artificial since it treats $x$ and $y$ differently, while the problem of agreement is really symmetric.

Coefficients like kappa or Krippendorffs' alpha are directly constructed for evaluating agreement, so is probably the most natural choice. Krippendorffs' alpha is the more general of the two, and  , indeed, it can be used with more than two raters. There is an R package irr which implements these and others. But these coefficients is only a number, it does not indicate the nature of the disagreement, so plots (specifically the Tukey mean-difference plot is very useful, also called Bland-Altman plot How does one interpret a Bland-Altman plot?). Regression models as above can also be useful in understanding the nature of the disagreement.

A book-length treatment is Kilem L. Gwet: "Handbook of interrater reliability" which I just got from the library for a project, so in some time maybe I will come back with some more information.

What you are looking for is agreement, not correlation. This is different: If comparing two sets of ratings from two raters (examiners), denote one set by $x$ the other $y$. If these ratings or scores satisfy $$ y=x+1, $$ say, then the correlation will be 1 but there is no agreement; in fact there is a constant disagreement of $|y-x| = 1$. Any measurement which does not take that into account cannot be an index of agreement! One correlation coefficient which could be used is the concordance correlation coefficient: see https://en.wikipedia.org/wiki/Concordance_correlation_coefficient This can be very close to some measures called ICC (there are many). t-test comparisons could be useful, but remember in the model $$ y_i = \alpha + \beta x_i + \epsilon_i $$ you want to test both $\alpha=0$ and $\beta=1$. This somehow, while it might be useful, seems artificial since it treats $x$ and $y$ differently, while the problem of agreement is really symmetric.

Coefficients like kappa or Krippendorff's alpha are directly constructed for evaluating agreement, so are probably the most natural choice. Krippendorff's alpha is the more general of the two, and, indeed, it can be used with more than two raters. There is an R package irr which implements these and others. But these coefficients are numbers and do not indicate the nature of the disagreement, so plots (specifically the Tukey mean-difference plot is very useful, also called Bland-Altman plot How does one interpret a Bland-Altman plot?). Regression models as above can also be useful in understanding the nature of the disagreement.

A book-length treatment is Kilem L. Gwet: "Handbook of interrater reliability" which I just got from the library for a project, so in some time maybe I will come back with some more information.

Source Link
kjetil b halvorsen
  • 82.8k
  • 32
  • 201
  • 663

What you are looking for is agreement not correlation. This is different: If comparing two sets of ratings from two raters (examiners), denote one set by $x$ the other $y$. If these ratings or scores satisfy $$ y=x+1, $$ say, then the correlation will be 1 but there is no agreement, if fact there is a constant disagreement of $|y-x| = 1$. Any measurement which do not take that into account cannot be an index of agreement! One correlation coefficient which could be used, is the concordance correlation coefficient, see https://en.wikipedia.org/wiki/Concordance_correlation_coefficient This can be very close to some measures called ICC (there are many). t-test comparisons could be useful, but remember in the model $$ y_i = \alpha + \beta x_i + \epsilon_i $$ you want to test both $\alpha=0$ and $\beta=1$. This somehow, while it might be useful, seems artificial since it treats $x$ and $y$ differently, while the problem of agreement is really symmetric.

Coefficients like kappa or Krippendorffs' alpha are directly constructed for evaluating agreement, so is probably the most natural choice. Krippendorffs' alpha is the more general of the two, and , indeed, it can be used with more than two raters. There is an R package irr which implements these and others. But these coefficients is only a number, it does not indicate the nature of the disagreement, so plots (specifically the Tukey mean-difference plot is very useful, also called Bland-Altman plot How does one interpret a Bland-Altman plot?). Regression models as above can also be useful in understanding the nature of the disagreement.

A book-length treatment is Kilem L. Gwet: "Handbook of interrater reliability" which I just got from the library for a project, so in some time maybe I will come back with some more information.