6
$\begingroup$

I am trying to build a model trained on binary labels that has a high precision for the top k predicted instances, and don’t care too much about recall or precision more generally. I was then interested in the idea of ranking instances in my classification problem rather than looking at prediction probabilities, so thought I could play around with XGBRanker with a single query group.

I noticed that Learning to Rank parameters can be passed to XGBClassifier without raising any errors, and in fact with a single query group XGBClassifier and XGBRanker seem to output the same results (see code below to reproduce in python with xgboost v2.0.3). Using XGBClassifier would be simpler here as it then doesn’t break sklearn compatibility, but I am unsure whether this is a correct usage.

I've asked about this on the XGBoost forum, but also wondered if anyone here had any insight into whether using XGBClassifier with objective='rank:map' is actually equivalent to using XGBRanker with a single query group.

    from sklearn.datasets import make_classification
    import numpy as np

    import xgboost as xgb

    # Make a synthetic ranking dataset for demonstration
    seed = 1994
    X, y = make_classification(random_state=seed)
    rng = np.random.default_rng(seed)
    n_query_groups = 1
    qid = rng.integers(0, n_query_groups, size=X.shape[0])

    # Sort the inputs based on query index
    sorted_idx = np.argsort(qid)
    X = X[sorted_idx, :]
    y = y[sorted_idx]
    qid_sorted = qid[sorted_idx]
    ranker = xgb.XGBRanker(lambdarank_num_pair_per_sample=8, objective="rank:map")
    ranker.fit(X, y, qid=qid_sorted)

    classif = xgb.XGBClassifier(lambdarank_num_pair_per_sample=8, objective="rank:map")
    classif.fit(X, y)

    rank_prediction = ranker.predict(X)
    classif_prediction = classif.predict(X)
    classif_prediction_proba = classif.predict_proba(X)[:, 1]

    assert np.array_equal(rank_prediction, classif_prediction_proba)
$\endgroup$

1 Answer 1

1
$\begingroup$

I've asked about this on the XGBoost forum, but also wondered if anyone here had any insight into whether using XGBClassifier with objective='rank:map' is actually equivalent to using XGBRanker with a single query group.

As per the documentation, the default objective for XGBRanker is rank:ndcg.

In this regard, XGBRanker and XGBClassifier need to have the same objective set for the results to be equivalent.

To illustrate this, consider that both XGBRanker and XGBClassifier are used to build models on a different dataset (Antonio, Almeida, Nunes, 2019) with the aim of predicting hotel cancellations (i.e. 1 if the customer cancels, 0 if not).

When running the below with no objective specified for XGBRanker and rank:map specified as the objective for XGBClassifier - an Assertion Error is yielded.

ranker = xgb.XGBRanker(lambdarank_num_pair_per_sample=8)
ranker.fit(x1, y, qid=qid_sorted)

classif = xgb.XGBClassifier(lambdarank_num_pair_per_sample=8, objective="rank:map")
classif.fit(x1, y)

rank_prediction = ranker.predict(x1)
classif_prediction = classif.predict(x1)
classif_prediction_proba = classif.predict_proba(x1)[:, 1]
assert np.array_equal(rank_prediction, classif_prediction_proba)

Additionally, np.array_equal(rank_prediction, classif_prediction_proba) returns a value of False. This also holds if the objective for XGBRanker is set to the default of rank:ndcg.

However, let us now specify the objective as rank:map for XGBRanker:

ranker = xgb.XGBRanker(lambdarank_num_pair_per_sample=8, objective="rank:map")

When XGBClassifier is set to the same objective, no Assertion Error is yielded and np.array_equal(rank_prediction, classif_prediction_proba) returns a value of True.

In this regard, XGBClassifier can be used to output the same results as XGBRanker - but the objective needs to be identical in both cases.

$\endgroup$
2
  • $\begingroup$ Thanks for your answer. I think in the first point you make, where both XGBRanker and XGBClassifier are given different objectives, I'm not surprised the Assertion Error is yielded as one would expect them to be doing different things. In my toy example, when you give XGBRanker and XGBClassifier the same arguments and objectives they seem to be outputting the same results; but what I really want to clarify is if in general using ranking objectives with XGBClassifier is the same as using XGBRanker with a single query group i.e. are the same methods and processes being evoked? $\endgroup$
    – A. Bollans
    Jan 22 at 10:58
  • $\begingroup$ I do not have sufficient knowledge to say definitively - but I can say that when I ran XGBRanker and XGBClassifer on the example mentioned above - ranker.predict and classif.predict_proba yielded the exact same predictions. $\endgroup$ Jan 23 at 19:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.