So I've got a dataset that can be ranked in two noisy ways $R_1$ and $R_2$. Let's call $R_1$ and $R_2$ functions from the dataset $x \in R^N$ to a real number between 0 and 1. Since they are rankings they never return the same value for two datapoints that are not exactly the same. So you could also think of these functions as imposing an ordering on my data set.
I've also got an oracle $R^*$ for a labeled training set that tells me the true rank, which is again a number between 0 and 1, but only for data points in my training set.
I don't care much about whether $R_1$ or $R_2$ actually return the true rank, I just want relative comparisons between ranks to be correct. That is, whenever
$R^*(x_1) < R^*(x_2)$
I would like
$R_1(x_1) < R_1(x_2)$ and $R_2(x_1) < R_2(x_2)$
Now, because $R_1$ and $R_2$ are noisy (I'm not sure what their noise model is, but it seems that $R_1$ gives me the correct comparison maybe 80% of the time, and $R_2$ gives me the correct comparison maybe 60% of the time, so better than pure chance but not great), this isn't always true.
Can I combine the outputs of $R_1$ and $R_2$ in some sensible way so that the result is better than using just one or the other?
So far I've just tried averaging $R_1$ and $R_2$, but that gets worse performance than just using the better of the two rankings.
I've also tried a weighted average $R_3(x) = 0.95 R_1(x) + 0.05 R_2(x)$ which got me about a 10% boost in performance over $R_1$.