I'm having a bit of trouble wrapping my head around something. I have a data set which contains two columns that essentially attempt to measure the same thing, one on a 1-50 continuous scale and another on a ordinal (5 levels) scale. Let's say they measure the risk of a comet impact (just an example). However the way the two columns are obtained is vastly different, the 1-50 scale is generated by an array of simulations, whereas the ordinal scale is based on expert opinion.
Now I want to know how much these two measurements "agree" with each other. So what I did was:
For the the continuous scale (call it $u$) I did a Kernel Density Estimation for each of the levels of the export opinion (call it $w \in W$). This estimates Probability Density Functions for each of the classes in $w$ given $u$. By Bayes rule I then computed the posterior probability $P(x|x \sim w_i)$ with $x$ being some sample (row) and $w_i$ some class of the expert opinion.
Now this sounded reasonable, to me. Except for choosing the priors (uniform, percentage in the original data, empirical?) and the density estimation (type of kernel, or is spline fitting better...)
But here it got iffy. Remember that I want to know agreement, and perhaps more importantly the samples on which the simulation and the expert disagree.
So for each of the samples I calculated all the posterior probabilities and then picked the maximum one. This allows me to calculate a Cohens Kappa for agreement and pick the rows that differ for further inspection (who was right?).
But this all seems a bit, well, wrong. What am I missing here?