Ordinal vs multinominal classification in XGboost: differences in one-hot encoding

Question

I have followed this post and tried to see if there will be any difference in predicted probabilities if I use different one-hot encoding in XGboost.

This is my code with some dummy data, which is actually not ordinal in nature.

from xgboost import XGBClassifier
from sklearn.datasets import load_iris
import pandas as pd

df = load_iris(as_frame=True)['frame']
x = df[[_ for _ in df.columns if _ != 'target']]
y = df['target']

model = XGBClassifier()

y1 = pd.get_dummies(y)
y2 = pd.get_dummies(y)
y2.loc[np.where(y==1)[0], [1,2]] = 1
y2.loc[np.where(y==2)[0], :] = 1

m1 = model.fit(x,y1)
m2 = model.fit(x,y2)

print(m1.predict_proba(x) == m2.predict_proba(x))

The predicted scores are exactly the same, so I am unsure if the difference in encoding does anything in XGBoost.

How can XGBoost handle ordinal classification then?

Answer 1

I think you are hitting undetermined behaviour here as XGBoost is not designed to have y be a pandas.DataFrame. I suspect that the sklearn API casts this into some numpy array but after than all bets are off. It appears it does something like one a classification task per column.

Strictly speaking XGBoost does not support ordinal regression, we can roughly approximate it to some degree by using multi-class classification as surrogate. i.e. having mlogloss (multi-class log-loss) as our evaluation metric and our objective as multi:softprob so we predict the probability of each data-point belonging to each class. Then we can directly change the predicted probabilities to ordinal values. The change itself can be done using two main options, without or with thresholding: A. We directly pick the class/label with the highest predicted probability (something akin to np.argmax(predicted_probs, axis=1) in Python) B. We apply thresholds to our predicted probability ranges to map them to ordinal values. (something akin to : np.argmax(predicted_probs > thresholds, axis=1) in Python).

The above being said, if we are dealing with multiple levels (e.g. 10+) and the spacing between is expected roughly equal, it might make sense to simply treat this as a regression problem directly. Similarly, if we are really aching about it, we can code the proportional odds loss (aka cumulative log-odds loss) as our objective manually and use that directly. Just remember we need the loss, the gradient and the Hessian.