So I've got a dataset that can be ranked in two noisy ways $R_1$ and $R_2$. Let's call $R_1$ and $R_2$ functions from the dataset $x \in R^N$ to a real number between 0 and 1. Since they are rankings they never return the same value for two datapoints that are not exactly the same. So you could also think of these functions as imposing an ordering on my data set.

I've also got an oracle $R^*$ for a labeled training set that tells me the true rank, which is again a number between 0 and 1, but only for data points in my training set.

I don't care much about whether $R_1$ or $R_2$ actually return the true rank, I just want relative comparisons between ranks to be correct. That is, whenever

$R^*(x_1) < R^*(x_2)$

I would like

$R_1(x_1) < R_1(x_2)$ and $R_2(x_1) < R_2(x_2)$

Now, because $R_1$ and $R_2$ are noisy (I'm not sure what their noise model is, but it seems that $R_1$ gives me the correct comparison maybe 80% of the time, and $R_2$ gives me the correct comparison maybe 60% of the time, so better than pure chance but not great), this isn't always true.

Can I combine the outputs of $R_1$ and $R_2$ in some sensible way so that the result is better than using just one or the other?


So far I've just tried averaging $R_1$ and $R_2$, but that gets worse performance than just using the better of the two rankings.


I've also tried a weighted average $R_3(x) = 0.95 R_1(x) + 0.05 R_2(x)$ which got me about a 10% boost in performance over $R_1$.


On the oracle-labelled dataset, you could treat the problem as a classification problem: you have four inputs, $R_1(x_1), R_1(x_2), R_2(x_1), R_2(x_2)$ and one binary valued output that takes value 1 whenever $R^*(x_1) < R^*(x_2)$?

You'd have far more data to train the classifier than the data that you've actually got (due to the combinatorial explosion), so it should be possible to fit a neural network or a GP or any other classifier to this really.

  • $\begingroup$ That's a good idea. I might actually want to train something that says "a < b" based on the output of R1 and R2 rather than something that goes directly from a and b to a score. I'll look into this. $\endgroup$ – mklingen Oct 23 '18 at 15:22
  • $\begingroup$ The reason for why I used the actual scores themselves instead of two binary valued outcomes, is to let the classification algorithm to have some flexibility - for example, if the difference between the rankings is very high in $R_1$ but not so much in $R_2$, this may "mean" something in the context of your problem. If the best classifier doesn't actually care about the magnitude of difference, it'd just fit something that ignores the magnitude. E.g. if you fit a neural net, the optimal choice of params might be to pass the $1 * R_1 - 1 * R_2$ scores through a sigmoid, recreating the 0/1 $\endgroup$ – InfProbSciX Oct 23 '18 at 16:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.