3
$\begingroup$

I have a set of ordered items A > B > C ... > F. For each element of the set I have a feature vector. Using these features I trained a neural network to predict the probability that A > B for any pair of items A and B. The neural network predictions are noisy. The output from the network may not be perfectly consistent with the true ranking.

My question is how do I go from having these pairwise ranking probabilities to a best-guess for the total ordering of the items in the set?

Improve this question
$\endgroup$
9
  • $\begingroup$ Do you wan't build NN classifier which return probability that the input vector classified to A and B? For example 85% that input vector is A and 15% that B $\endgroup$
    – itdxer
    May 13, 2015 at 8:45
  • $\begingroup$ No. I have a vector for A and B and I have also a NN that tells me probability that A > B. I want to go from this to a global ranking. $\endgroup$
    – Aaron
    May 13, 2015 at 14:27
  • $\begingroup$ But why you didn't use simple comparison between vectors and get 100% result? $\endgroup$
    – itdxer
    May 13, 2015 at 14:34
  • $\begingroup$ What do you mean simple comparison between vectors? There is no simple way from looking at the feature vectors to tell if item A should be ranked higher or lower than item B. That's why I need a NN to predict it. The question I'm asking is how to go from having these pair-wise predictions to a total ordering across a larger set of items. $\endgroup$
    – Aaron
    May 13, 2015 at 17:12
  • 1
    $\begingroup$ Are you sure that there is transitivity in your rankings? Or might this be like the "rock, paper, scissors" game where A > B and B > C but also C > A? In that case, there is no simple ranking of all items. $\endgroup$
    – EdM
    May 13, 2015 at 17:37

2 Answers 2

Reset to default
1
$\begingroup$

If you have sufficiently few items to do an exhaustive search of all orderings, decide on a criterion for what a "best guess" means for the overall ordering, and report the ordering that maximizes your criterion. For example, a simple "best-guess" criterion could be the number of A:B pairs that were called in the correct order (A>B vs B>A) by the neural network, regardless of the probability values. (The proportion of correct pairwise rankings is the basis of the area under the curve, AUC, criterion for evaluating receiver-operating-characteristic curves.) Or you might want to add a criterion that more strongly weights the neural-network calls made with high probability. But you would have to choose such weights, depending on what you mean by "best guess."

Improve this answer
$\endgroup$
0
$\begingroup$

Assuming that pair-wise rank means a series of binary values - either 0 or 1 depending on whether the team won or lost (is better or worse) against another team and there are a multiple such binary outcomes for different teams, then a overall ranking can be built using the Bradley Terry model which will convert these series of binary scores to a real number which can then be compared against one another. The team with the higher value is ranked higher.

The advantage of these type of models is that there need not be n(n-1)/2 comparisons. As long as all teams are sampled enough times, a tiny subset is enough to create a reasonable ranking list. In your case, since the results are noisy (and also automatically generated you may want to consider more combinations. Alternatively strive for higher accuracy in your model (maybe even manually classifying a few samples) and then even very few pair-wise results will help generate a reasonable scores

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.