5
$\begingroup$

I am trying to build a ranking model using xgboost, which seems to work, but am not sure however of how to interpret the predictions. I haven't been able to find relevant documentation or examples on this particular task, so I am unsure if I'm either failing to correctly build a ranking model, which gives nonsensical output, or if I'm just not able to make sense of it.

This is what I've tried so far:

library(xgboost)

#generate some random data and target
train <- matrix(sample(c(runif(100), rep(0, 100)), 2000, replace = T), ncol = 20)
target <- sample(c(0, 1, 2), 100, replace = T)
test <- matrix(sample(c(runif(100), rep(0, 100)), 1000, replace = T), ncol = 20)

xgbTrain <- xgb.DMatrix(train, label = target)
xgbTest <- xgb.DMatrix(test)

params <- list(booster = 'gbtree',
              objective = 'rank:pairwise')

rankModel <- xgb.train(params, xgbTrain, 10, watchlist = list(tr = xgbTrain), eval_metric = 'ndcg')

print(predict(rankModel, xgbTest))

Which prints:

 [1] -0.94882607 -0.58704734  1.64421082  1.75749838 -0.82614720  0.30855450 -0.23618424  2.03426790  0.18326324  1.54318023  1.60956442 -0.26394469
 [13]  2.18902779 -0.82873511  2.01967430 -0.55936623 -0.77433181 -0.95465362  0.30088556  2.07671452 -0.98471618 -0.55923200 -0.96335471 -0.64822674
 [25]  1.62699771 -0.78486574 -0.80902183 -1.09310806 -0.75859952  0.76334584  0.64056098  0.48438376  0.48405507  1.67765403  0.63187867  0.66793621
 [37]  1.00044215 -0.87645602  1.60261607  0.56360698 -1.14934421  1.68098438  2.03522825 -0.95139503 -1.02189994 -0.94723392 -0.69983196 -1.20132697
 [49]  1.26217628  0.79300237  0.79510546  2.24825048  1.84355474  1.10526133 -1.37557507 -1.00994503  1.77173507  1.13348961  0.79858458 -0.67265379
 [61]  1.01461124  0.48507297 -0.83393693  1.67525303  1.24216270  0.02764088  0.91251671  2.04108095  2.17401123 -0.62811875  0.73657835  2.39004827
 [73] -0.18949705  0.60755765  2.26420307 -1.23398590  0.59091914  1.65885520 -0.30570436  2.00285411  0.67023963  2.06346416  1.69319844  2.27233768
 [85]  0.32537588  2.29775119  0.08256459 -1.16774368  0.78462589 -1.01925206 -0.81494403  0.50675780 -0.65493345 -0.17356467 -0.76901770 -0.86279511
 [97] -1.08725667  1.94247663 -0.96650219  0.88204288
$\endgroup$
  • $\begingroup$ It looks like you're missing target values for your test set so it will be difficult for you to assess the predictions there. However, you can look at predictions on the training set against the true target values to get a sense of what the model is doing. plot(target ~ predict(rankModel, xgbTrain)) should give you some good insight. $\endgroup$ – assumednormal Nov 28 '15 at 16:01
  • 2
    $\begingroup$ What I don't understand is that I would be supposed to get a ranking through "predict". Is that not the case? Here I get predictions, that might be accurate if rounded, but that only contains information on the "first"of the ranking. $\endgroup$ – Gerome Pistre Nov 29 '15 at 0:32
1
$\begingroup$

I think you need to do 2 things:

  • set the group info correctly so that all documents belonging to a query are ranked in the same round
  • you need to write a rerank function which will reorder the results for each query by these scores in decreasing order. These are not probabilities, just prediction scores. If you need probabilities, you can renormalize the scores.
| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.