# Can two imperfect rankings be combined to produce a better ranking?

So I've got a dataset that can be ranked in two noisy ways $$R_1$$ and $$R_2$$. Let's call $$R_1$$ and $$R_2$$ functions from the dataset $$x \in R^N$$ to a real number between 0 and 1. Since they are rankings they never return the same value for two datapoints that are not exactly the same. So you could also think of these functions as imposing an ordering on my data set.

I've also got an oracle $$R^*$$ for a labeled training set that tells me the true rank, which is again a number between 0 and 1, but only for data points in my training set.

I don't care much about whether $$R_1$$ or $$R_2$$ actually return the true rank, I just want relative comparisons between ranks to be correct. That is, whenever

$$R^*(x_1) < R^*(x_2)$$

I would like

$$R_1(x_1) < R_1(x_2)$$ and $$R_2(x_1) < R_2(x_2)$$

Now, because $$R_1$$ and $$R_2$$ are noisy (I'm not sure what their noise model is, but it seems that $$R_1$$ gives me the correct comparison maybe 80% of the time, and $$R_2$$ gives me the correct comparison maybe 60% of the time, so better than pure chance but not great), this isn't always true.

Can I combine the outputs of $$R_1$$ and $$R_2$$ in some sensible way so that the result is better than using just one or the other?

EDIT:

So far I've just tried averaging $$R_1$$ and $$R_2$$, but that gets worse performance than just using the better of the two rankings.

EDIT 2:

I've also tried a weighted average $$R_3(x) = 0.95 R_1(x) + 0.05 R_2(x)$$ which got me about a 10% boost in performance over $$R_1$$.

On the oracle-labelled dataset, you could treat the problem as a classification problem: you have four inputs, $$R_1(x_1), R_1(x_2), R_2(x_1), R_2(x_2)$$ and one binary valued output that takes value 1 whenever $$R^*(x_1) < R^*(x_2)$$?
• The reason for why I used the actual scores themselves instead of two binary valued outcomes, is to let the classification algorithm to have some flexibility - for example, if the difference between the rankings is very high in $R_1$ but not so much in $R_2$, this may "mean" something in the context of your problem. If the best classifier doesn't actually care about the magnitude of difference, it'd just fit something that ignores the magnitude. E.g. if you fit a neural net, the optimal choice of params might be to pass the $1 * R_1 - 1 * R_2$ scores through a sigmoid, recreating the 0/1 – InfProbSciX Oct 23 '18 at 16:11