Calculating agreement between an ordinal and continuous scale

I'm having a bit of trouble wrapping my head around something. I have a data set which contains two columns that essentially attempt to measure the same thing, one on a 1-50 continuous scale and another on a ordinal (5 levels) scale. Let's say they measure the risk of a comet impact (just an example). However the way the two columns are obtained is vastly different, the 1-50 scale is generated by an array of simulations, whereas the ordinal scale is based on expert opinion.

Now I want to know how much these two measurements "agree" with each other. So what I did was:

For the the continuous scale (call it $u$) I did a Kernel Density Estimation for each of the levels of the export opinion (call it $w \in W$). This estimates Probability Density Functions for each of the classes in $w$ given $u$. By Bayes rule I then computed the posterior probability $P(x|x \sim w_i)$ with $x$ being some sample (row) and $w_i$ some class of the expert opinion.

Now this sounded reasonable, to me. Except for choosing the priors (uniform, percentage in the original data, empirical?) and the density estimation (type of kernel, or is spline fitting better...)

But here it got iffy. Remember that I want to know agreement, and perhaps more importantly the samples on which the simulation and the expert disagree.

So for each of the samples I calculated all the posterior probabilities and then picked the maximum one. This allows me to calculate a Cohens Kappa for agreement and pick the rows that differ for further inspection (who was right?).

But this all seems a bit, well, wrong. What am I missing here?

• This sounds like a very complicated approach. Why not start with a simple Pearson correlation? (Since the second column has only five possible values it'll be far from perfect, but it's easily interpretable). A second option is probit regression of the ordinal data on the continuous data. Am I missing something with these two options? Feb 27 '14 at 21:03
• @DavidRobinson the data cannot be assumed normally distributed. And the there must be a mapping between all of them, so I can't do binary (with probit, if I understood correctly) Feb 27 '14 at 21:10
• Why do you think normally distributed data to use a Pearson correlation? Feb 27 '14 at 22:17
• Probit would work fine, I'm not sure I understand what you mean by mapping. And you can use spearman if you're concerned about normality of the continuous data. Feb 27 '14 at 22:32

• rank correlation will look at the ordering of $u$ and $w$. in that regard $u$ will be treated as $w$, i.e. like ordinal data. Feb 27 '14 at 22:17