2
$\begingroup$

I'm having a bit of trouble wrapping my head around something. I have a data set which contains two columns that essentially attempt to measure the same thing, one on a 1-50 continuous scale and another on a ordinal (5 levels) scale. Let's say they measure the risk of a comet impact (just an example). However the way the two columns are obtained is vastly different, the 1-50 scale is generated by an array of simulations, whereas the ordinal scale is based on expert opinion.

Now I want to know how much these two measurements "agree" with each other. So what I did was:

For the the continuous scale (call it $u$) I did a Kernel Density Estimation for each of the levels of the export opinion (call it $w \in W$). This estimates Probability Density Functions for each of the classes in $w$ given $u$. By Bayes rule I then computed the posterior probability $P(x|x \sim w_i)$ with $x$ being some sample (row) and $w_i$ some class of the expert opinion.

Now this sounded reasonable, to me. Except for choosing the priors (uniform, percentage in the original data, empirical?) and the density estimation (type of kernel, or is spline fitting better...)

But here it got iffy. Remember that I want to know agreement, and perhaps more importantly the samples on which the simulation and the expert disagree.

So for each of the samples I calculated all the posterior probabilities and then picked the maximum one. This allows me to calculate a Cohens Kappa for agreement and pick the rows that differ for further inspection (who was right?).

But this all seems a bit, well, wrong. What am I missing here?

Improve this question
$\endgroup$
4
  • $\begingroup$ This sounds like a very complicated approach. Why not start with a simple Pearson correlation? (Since the second column has only five possible values it'll be far from perfect, but it's easily interpretable). A second option is probit regression of the ordinal data on the continuous data. Am I missing something with these two options? $\endgroup$
    – David Robinson
    Feb 27 '14 at 21:03
  • $\begingroup$ @DavidRobinson the data cannot be assumed normally distributed. And the there must be a mapping between all of them, so I can't do binary (with probit, if I understood correctly) $\endgroup$
    – JoelKuiper
    Feb 27 '14 at 21:10
  • 1
    $\begingroup$ Why do you think normally distributed data to use a Pearson correlation? $\endgroup$
    – Glen_b
    Feb 27 '14 at 22:17
  • $\begingroup$ Probit would work fine, I'm not sure I understand what you mean by mapping. And you can use spearman if you're concerned about normality of the continuous data. $\endgroup$
    – David Robinson
    Feb 27 '14 at 22:32
4
$\begingroup$

Kendal tau rank correlation coefficient should be appropriate here, as well as Spearman's correlation

Improve this answer
$\endgroup$
3
  • $\begingroup$ Do you have an example on how to do this? At first glance it doesn't seem to be appropriate for comparing ordinal and continuous data. $\endgroup$
    – JoelKuiper
    Feb 27 '14 at 22:13
  • 2
    $\begingroup$ Joel, Kendall's tau requires only that you be able to order each variable, so it applies here. I have some concerns that it might not be a very powerful procedure (especially because of all the ties it will have to cope with)--but that means if you apply it and it shows a clear and significant difference, then that difference would likely be even more significant with a better test. $\endgroup$
    – whuber ♦
    Feb 27 '14 at 22:17
  • $\begingroup$ rank correlation will look at the ordering of $u$ and $w$. in that regard $u$ will be treated as $w$, i.e. like ordinal data. $\endgroup$
    – Aksakal
    Feb 27 '14 at 22:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.